[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251025 20:02:37.072734 29443 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.192.254:33913
I20251025 20:02:37.075048 29443 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:02:37.076596 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:02:37.100553 29452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:02:37.102197 29450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:02:37.100582 29449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:02:38.360148 29451 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1257 milliseconds
I20251025 20:02:38.360324 29443 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251025 20:02:38.361855 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:02:38.362187 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:02:38.362393 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422558362365 us; error 0 us; skew 500 ppm
I20251025 20:02:38.363339 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:02:38.372542 29443 webserver.cc:492] Webserver started at http://127.28.192.254:43435/ using document root <none> and password file <none>
I20251025 20:02:38.373783 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:02:38.374152 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:02:38.374714 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:02:38.381965 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/master-0-root/instance:
uuid: "7857dfba33b246c99d5979bb2a34a369"
format_stamp: "Formatted at 2025-10-25 20:02:38 on dist-test-slave-svnn"
I20251025 20:02:38.392122 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20251025 20:02:38.400274 29458 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:02:38.402168 29443 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.005s
I20251025 20:02:38.402710 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/master-0-root
uuid: "7857dfba33b246c99d5979bb2a34a369"
format_stamp: "Formatted at 2025-10-25 20:02:38 on dist-test-slave-svnn"
I20251025 20:02:38.403167 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:02:38.469295 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:02:38.471339 29443 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:02:38.472121 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:02:38.590131 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.254:33913
I20251025 20:02:38.590238 29509 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.254:33913 every 8 connection(s)
I20251025 20:02:38.602872 29510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:38.632400 29510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369: Bootstrap starting.
I20251025 20:02:38.641486 29510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:38.644287 29510 log.cc:826] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369: Log is configured to *not* fsync() on all Append() calls
I20251025 20:02:38.651299 29510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369: No bootstrap required, opened a new log
I20251025 20:02:38.678529 29510 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857dfba33b246c99d5979bb2a34a369" member_type: VOTER }
I20251025 20:02:38.679220 29510 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:38.679553 29510 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7857dfba33b246c99d5979bb2a34a369, State: Initialized, Role: FOLLOWER
I20251025 20:02:38.680598 29510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857dfba33b246c99d5979bb2a34a369" member_type: VOTER }
I20251025 20:02:38.681212 29510 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:02:38.681574 29510 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:02:38.682019 29510 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:38.688251 29510 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857dfba33b246c99d5979bb2a34a369" member_type: VOTER }
I20251025 20:02:38.689302 29510 leader_election.cc:304] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7857dfba33b246c99d5979bb2a34a369; no voters:
I20251025 20:02:38.692188 29510 leader_election.cc:290] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:02:38.692600 29513 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:02:38.695815 29513 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 1 LEADER]: Becoming Leader. State: Replica: 7857dfba33b246c99d5979bb2a34a369, State: Running, Role: LEADER
I20251025 20:02:38.696913 29513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857dfba33b246c99d5979bb2a34a369" member_type: VOTER }
I20251025 20:02:38.698745 29510 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:02:38.717712 29515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7857dfba33b246c99d5979bb2a34a369. Latest consensus state: current_term: 1 leader_uuid: "7857dfba33b246c99d5979bb2a34a369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857dfba33b246c99d5979bb2a34a369" member_type: VOTER } }
I20251025 20:02:38.718006 29514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7857dfba33b246c99d5979bb2a34a369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857dfba33b246c99d5979bb2a34a369" member_type: VOTER } }
I20251025 20:02:38.718870 29515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [sys.catalog]: This master's current role is: LEADER
I20251025 20:02:38.719054 29514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [sys.catalog]: This master's current role is: LEADER
I20251025 20:02:38.725276 29522 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:02:38.746538 29522 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:02:38.748929 29443 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251025 20:02:38.771688 29522 catalog_manager.cc:1357] Generated new cluster ID: 7bf17cfad2444c199193a7cc0915b2f6
I20251025 20:02:38.771993 29522 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:02:38.798835 29522 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:02:38.800683 29522 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:02:38.819586 29522 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369: Generated new TSK 0
I20251025 20:02:38.822000 29522 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 20:02:38.882900 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:02:38.896651 29532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:02:38.898015 29533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:02:38.904999 29443 server_base.cc:1047] running on GCE node
W20251025 20:02:38.905872 29535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:02:38.907094 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:02:38.907337 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:02:38.907527 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422558907496 us; error 0 us; skew 500 ppm
I20251025 20:02:38.908178 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:02:38.915684 29443 webserver.cc:492] Webserver started at http://127.28.192.193:40707/ using document root <none> and password file <none>
I20251025 20:02:38.916406 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:02:38.916651 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:02:38.916971 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:02:38.918515 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-0-root/instance:
uuid: "23b530a2476d464eb636e2f3ea2e1442"
format_stamp: "Formatted at 2025-10-25 20:02:38 on dist-test-slave-svnn"
I20251025 20:02:38.927284 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20251025 20:02:38.933748 29540 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:02:38.934955 29443 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20251025 20:02:38.935444 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-0-root
uuid: "23b530a2476d464eb636e2f3ea2e1442"
format_stamp: "Formatted at 2025-10-25 20:02:38 on dist-test-slave-svnn"
I20251025 20:02:38.935894 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:02:38.963021 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:02:38.965222 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:02:38.969786 29443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:02:38.976771 29443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:02:38.977167 29443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:02:38.977790 29443 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:02:38.978178 29443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.002s sys 0.000s
I20251025 20:02:39.124619 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.193:36283
I20251025 20:02:39.124713 29602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.193:36283 every 8 connection(s)
I20251025 20:02:39.136912 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:02:39.164254 29607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:02:39.165612 29608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:02:39.172389 29610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:02:39.177485 29443 server_base.cc:1047] running on GCE node
I20251025 20:02:39.178915 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:02:39.179275 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:02:39.179533 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422559179506 us; error 0 us; skew 500 ppm
I20251025 20:02:39.180413 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:02:39.184336 29443 webserver.cc:492] Webserver started at http://127.28.192.194:44349/ using document root <none> and password file <none>
I20251025 20:02:39.185217 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:02:39.185501 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:02:39.185926 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:02:39.188027 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-1-root/instance:
uuid: "e76383577c9d4685bf6d452bf2e7ec32"
format_stamp: "Formatted at 2025-10-25 20:02:39 on dist-test-slave-svnn"
I20251025 20:02:39.190775 29603 heartbeater.cc:344] Connected to a master server at 127.28.192.254:33913
I20251025 20:02:39.191511 29603 heartbeater.cc:461] Registering TS with master...
I20251025 20:02:39.192749 29603 heartbeater.cc:507] Master 127.28.192.254:33913 requested a full tablet report, sending...
I20251025 20:02:39.196408 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20251025 20:02:39.197052 29475 ts_manager.cc:194] Registered new tserver with Master: 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283)
I20251025 20:02:39.200958 29475 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43516
I20251025 20:02:39.203603 29615 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:02:39.205157 29443 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20251025 20:02:39.205595 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-1-root
uuid: "e76383577c9d4685bf6d452bf2e7ec32"
format_stamp: "Formatted at 2025-10-25 20:02:39 on dist-test-slave-svnn"
I20251025 20:02:39.206075 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:02:39.228490 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:02:39.230639 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:02:39.232553 29443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:02:39.236832 29443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:02:39.237160 29443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:02:39.237442 29443 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:02:39.237648 29443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:02:39.369482 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.194:43239
I20251025 20:02:39.369550 29677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.194:43239 every 8 connection(s)
I20251025 20:02:39.379760 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251025 20:02:39.396251 29678 heartbeater.cc:344] Connected to a master server at 127.28.192.254:33913
W20251025 20:02:39.396307 29682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:02:39.396982 29678 heartbeater.cc:461] Registering TS with master...
W20251025 20:02:39.398659 29681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:02:39.399228 29678 heartbeater.cc:507] Master 127.28.192.254:33913 requested a full tablet report, sending...
I20251025 20:02:39.403373 29475 ts_manager.cc:194] Registered new tserver with Master: e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
W20251025 20:02:39.405373 29684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:02:39.406033 29475 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43530
I20251025 20:02:39.409883 29443 server_base.cc:1047] running on GCE node
I20251025 20:02:39.411077 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:02:39.411377 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:02:39.411597 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422559411577 us; error 0 us; skew 500 ppm
I20251025 20:02:39.412389 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:02:39.415622 29443 webserver.cc:492] Webserver started at http://127.28.192.195:36051/ using document root <none> and password file <none>
I20251025 20:02:39.416347 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:02:39.416606 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:02:39.416949 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:02:39.418529 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-2-root/instance:
uuid: "10924e710604471a813b87ff5bbcfb8f"
format_stamp: "Formatted at 2025-10-25 20:02:39 on dist-test-slave-svnn"
I20251025 20:02:39.425200 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20251025 20:02:39.430402 29689 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:02:39.431689 29443 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:02:39.432231 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-2-root
uuid: "10924e710604471a813b87ff5bbcfb8f"
format_stamp: "Formatted at 2025-10-25 20:02:39 on dist-test-slave-svnn"
I20251025 20:02:39.432651 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761422556872874-29443-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:02:39.469274 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:02:39.471186 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:02:39.474354 29443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:02:39.479346 29443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:02:39.479693 29443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:02:39.480000 29443 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:02:39.480206 29443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:02:39.633296 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.195:45807
I20251025 20:02:39.633344 29751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.195:45807 every 8 connection(s)
I20251025 20:02:39.657475 29752 heartbeater.cc:344] Connected to a master server at 127.28.192.254:33913
I20251025 20:02:39.657969 29752 heartbeater.cc:461] Registering TS with master...
I20251025 20:02:39.659061 29752 heartbeater.cc:507] Master 127.28.192.254:33913 requested a full tablet report, sending...
I20251025 20:02:39.661505 29475 ts_manager.cc:194] Registered new tserver with Master: 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:39.662194 29443 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016977547s
I20251025 20:02:39.663492 29475 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43540
I20251025 20:02:39.667131 29443 test_util.cc:276] Using random seed: 637002641
I20251025 20:02:39.738159 29475 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43550:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:02:39.744066 29475 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:02:39.903172 29717 tablet_service.cc:1505] Processing CreateTablet for tablet 7c443311caa74ff4a1df53c5323f21b2 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251025 20:02:39.910027 29716 tablet_service.cc:1505] Processing CreateTablet for tablet c43e53a8c2594107bb43f93727e95284 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251025 20:02:39.919747 29717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c443311caa74ff4a1df53c5323f21b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.911155 29714 tablet_service.cc:1505] Processing CreateTablet for tablet 585e4a99f17f4f0e9b012f6cc74bdc99 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251025 20:02:39.925738 29643 tablet_service.cc:1505] Processing CreateTablet for tablet 7c443311caa74ff4a1df53c5323f21b2 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251025 20:02:39.918932 29715 tablet_service.cc:1505] Processing CreateTablet for tablet 9893e4b3767a41909e80e29b83fce0b0 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251025 20:02:39.923420 29642 tablet_service.cc:1505] Processing CreateTablet for tablet c43e53a8c2594107bb43f93727e95284 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251025 20:02:39.934089 29716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c43e53a8c2594107bb43f93727e95284. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.931957 29640 tablet_service.cc:1505] Processing CreateTablet for tablet 585e4a99f17f4f0e9b012f6cc74bdc99 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251025 20:02:39.934222 29713 tablet_service.cc:1505] Processing CreateTablet for tablet fb7a820773d0484caab2a0e7a7bb9e38 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251025 20:02:39.940207 29639 tablet_service.cc:1505] Processing CreateTablet for tablet fb7a820773d0484caab2a0e7a7bb9e38 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251025 20:02:39.941956 29568 tablet_service.cc:1505] Processing CreateTablet for tablet 7c443311caa74ff4a1df53c5323f21b2 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251025 20:02:39.942726 29643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c443311caa74ff4a1df53c5323f21b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.943184 29641 tablet_service.cc:1505] Processing CreateTablet for tablet 9893e4b3767a41909e80e29b83fce0b0 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251025 20:02:39.951617 29568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c443311caa74ff4a1df53c5323f21b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.948873 29714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585e4a99f17f4f0e9b012f6cc74bdc99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.947099 29567 tablet_service.cc:1505] Processing CreateTablet for tablet c43e53a8c2594107bb43f93727e95284 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251025 20:02:39.957384 29639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb7a820773d0484caab2a0e7a7bb9e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.959614 29565 tablet_service.cc:1505] Processing CreateTablet for tablet 585e4a99f17f4f0e9b012f6cc74bdc99 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251025 20:02:39.961741 29715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9893e4b3767a41909e80e29b83fce0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.959599 29567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c43e53a8c2594107bb43f93727e95284. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.964298 29640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585e4a99f17f4f0e9b012f6cc74bdc99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.958302 29638 tablet_service.cc:1505] Processing CreateTablet for tablet 4a12402079514257b1bd8b657887b2bc (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251025 20:02:39.964984 29564 tablet_service.cc:1505] Processing CreateTablet for tablet fb7a820773d0484caab2a0e7a7bb9e38 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251025 20:02:39.958299 29566 tablet_service.cc:1505] Processing CreateTablet for tablet 9893e4b3767a41909e80e29b83fce0b0 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251025 20:02:39.967911 29713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb7a820773d0484caab2a0e7a7bb9e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.968554 29565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585e4a99f17f4f0e9b012f6cc74bdc99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.976091 29563 tablet_service.cc:1505] Processing CreateTablet for tablet 4a12402079514257b1bd8b657887b2bc (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251025 20:02:39.980994 29641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9893e4b3767a41909e80e29b83fce0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.984910 29564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb7a820773d0484caab2a0e7a7bb9e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.985918 29712 tablet_service.cc:1505] Processing CreateTablet for tablet 4a12402079514257b1bd8b657887b2bc (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251025 20:02:39.988588 29712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a12402079514257b1bd8b657887b2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.990293 29638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a12402079514257b1bd8b657887b2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:39.994230 29562 tablet_service.cc:1505] Processing CreateTablet for tablet 9ae63ee9cca14dae8be38ff2b05c01a6 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251025 20:02:39.999414 29566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9893e4b3767a41909e80e29b83fce0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.002812 29637 tablet_service.cc:1505] Processing CreateTablet for tablet 9ae63ee9cca14dae8be38ff2b05c01a6 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251025 20:02:40.033370 29711 tablet_service.cc:1505] Processing CreateTablet for tablet 9ae63ee9cca14dae8be38ff2b05c01a6 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251025 20:02:40.035706 29711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ae63ee9cca14dae8be38ff2b05c01a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.041713 29562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ae63ee9cca14dae8be38ff2b05c01a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.049010 29642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c43e53a8c2594107bb43f93727e95284. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.054458 29637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ae63ee9cca14dae8be38ff2b05c01a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.076186 29561 tablet_service.cc:1505] Processing CreateTablet for tablet 1cf1641f71ef4ece8e154d1dba151cea (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251025 20:02:40.082993 29563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a12402079514257b1bd8b657887b2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.092423 29561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf1641f71ef4ece8e154d1dba151cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.102176 29636 tablet_service.cc:1505] Processing CreateTablet for tablet 1cf1641f71ef4ece8e154d1dba151cea (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251025 20:02:40.112416 29710 tablet_service.cc:1505] Processing CreateTablet for tablet 1cf1641f71ef4ece8e154d1dba151cea (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251025 20:02:40.117496 29710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf1641f71ef4ece8e154d1dba151cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.113124 29636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf1641f71ef4ece8e154d1dba151cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.122998 29560 tablet_service.cc:1505] Processing CreateTablet for tablet b067857e85d640b584839f045b12b019 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251025 20:02:40.134362 29560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b067857e85d640b584839f045b12b019. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.185634 29635 tablet_service.cc:1505] Processing CreateTablet for tablet b067857e85d640b584839f045b12b019 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251025 20:02:40.187223 29635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b067857e85d640b584839f045b12b019. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.200949 29709 tablet_service.cc:1505] Processing CreateTablet for tablet b067857e85d640b584839f045b12b019 (DEFAULT_TABLE table=test-workload [id=79222d595a5b4f438c7bc4a71ef23d0d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251025 20:02:40.202584 29709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b067857e85d640b584839f045b12b019. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:02:40.207051 29603 heartbeater.cc:499] Master 127.28.192.254:33913 was elected leader, sending a full tablet report...
I20251025 20:02:40.270483 29770 tablet_bootstrap.cc:492] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32: Bootstrap starting.
I20251025 20:02:40.286060 29771 tablet_bootstrap.cc:492] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442: Bootstrap starting.
I20251025 20:02:40.298149 29772 tablet_bootstrap.cc:492] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f: Bootstrap starting.
I20251025 20:02:40.372051 29770 tablet_bootstrap.cc:654] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.378545 29772 tablet_bootstrap.cc:654] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.381536 29771 tablet_bootstrap.cc:654] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.397156 29770 tablet_bootstrap.cc:492] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32: No bootstrap required, opened a new log
I20251025 20:02:40.397966 29770 ts_tablet_manager.cc:1403] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32: Time spent bootstrapping tablet: real 0.133s user 0.004s sys 0.018s
I20251025 20:02:40.399852 29771 tablet_bootstrap.cc:492] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442: No bootstrap required, opened a new log
I20251025 20:02:40.403014 29770 raft_consensus.cc:359] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.405853 29770 raft_consensus.cc:385] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.405439 29771 ts_tablet_manager.cc:1403] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.009s
I20251025 20:02:40.406441 29770 raft_consensus.cc:740] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.407665 29772 tablet_bootstrap.cc:492] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f: No bootstrap required, opened a new log
I20251025 20:02:40.407397 29770 consensus_queue.cc:260] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.408401 29772 ts_tablet_manager.cc:1403] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f: Time spent bootstrapping tablet: real 0.123s user 0.026s sys 0.000s
I20251025 20:02:40.409678 29678 heartbeater.cc:499] Master 127.28.192.254:33913 was elected leader, sending a full tablet report...
I20251025 20:02:40.409739 29771 raft_consensus.cc:359] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:40.410611 29771 raft_consensus.cc:385] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.410945 29771 raft_consensus.cc:740] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.411854 29771 consensus_queue.cc:260] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:40.411878 29772 raft_consensus.cc:359] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.413242 29772 raft_consensus.cc:385] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.413671 29772 raft_consensus.cc:740] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.415411 29772 consensus_queue.cc:260] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.418440 29770 ts_tablet_manager.cc:1434] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32: Time spent starting tablet: real 0.020s user 0.010s sys 0.000s
I20251025 20:02:40.419773 29770 tablet_bootstrap.cc:492] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32: Bootstrap starting.
I20251025 20:02:40.425944 29772 ts_tablet_manager.cc:1434] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f: Time spent starting tablet: real 0.017s user 0.003s sys 0.010s
I20251025 20:02:40.427058 29752 heartbeater.cc:499] Master 127.28.192.254:33913 was elected leader, sending a full tablet report...
I20251025 20:02:40.428927 29770 tablet_bootstrap.cc:654] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.434782 29772 tablet_bootstrap.cc:492] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f: Bootstrap starting.
I20251025 20:02:40.445725 29772 tablet_bootstrap.cc:654] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.443307 29771 ts_tablet_manager.cc:1434] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442: Time spent starting tablet: real 0.037s user 0.010s sys 0.000s
I20251025 20:02:40.448207 29771 tablet_bootstrap.cc:492] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442: Bootstrap starting.
I20251025 20:02:40.449231 29776 raft_consensus.cc:493] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:40.449851 29776 raft_consensus.cc:515] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.453461 29776 leader_election.cc:290] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:40.464493 29770 tablet_bootstrap.cc:492] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32: No bootstrap required, opened a new log
I20251025 20:02:40.467046 29770 ts_tablet_manager.cc:1403] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32: Time spent bootstrapping tablet: real 0.048s user 0.017s sys 0.019s
I20251025 20:02:40.465328 29771 tablet_bootstrap.cc:654] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.472496 29770 raft_consensus.cc:359] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.478153 29770 raft_consensus.cc:385] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.479338 29770 raft_consensus.cc:740] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.481133 29770 consensus_queue.cc:260] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.483747 29770 ts_tablet_manager.cc:1434] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32: Time spent starting tablet: real 0.015s user 0.006s sys 0.002s
I20251025 20:02:40.486220 29770 tablet_bootstrap.cc:492] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32: Bootstrap starting.
I20251025 20:02:40.494285 29771 tablet_bootstrap.cc:492] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442: No bootstrap required, opened a new log
I20251025 20:02:40.495720 29770 tablet_bootstrap.cc:654] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.496079 29771 ts_tablet_manager.cc:1403] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.002s
I20251025 20:02:40.515832 29771 raft_consensus.cc:359] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.517042 29771 raft_consensus.cc:385] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.518013 29777 raft_consensus.cc:493] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:40.517987 29771 raft_consensus.cc:740] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.521147 29777 raft_consensus.cc:515] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:40.524468 29771 consensus_queue.cc:260] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.527130 29777 leader_election.cc:290] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:40.534617 29772 tablet_bootstrap.cc:492] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f: No bootstrap required, opened a new log
I20251025 20:02:40.535209 29772 ts_tablet_manager.cc:1403] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f: Time spent bootstrapping tablet: real 0.101s user 0.029s sys 0.014s
I20251025 20:02:40.557885 29770 tablet_bootstrap.cc:492] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32: No bootstrap required, opened a new log
I20251025 20:02:40.558764 29770 ts_tablet_manager.cc:1403] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32: Time spent bootstrapping tablet: real 0.074s user 0.017s sys 0.005s
I20251025 20:02:40.563833 29772 raft_consensus.cc:359] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.566937 29772 raft_consensus.cc:385] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.570396 29772 raft_consensus.cc:740] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.577142 29772 consensus_queue.cc:260] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.577859 29771 ts_tablet_manager.cc:1434] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442: Time spent starting tablet: real 0.080s user 0.001s sys 0.030s
I20251025 20:02:40.576560 29770 raft_consensus.cc:359] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:40.579700 29770 raft_consensus.cc:385] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.580215 29771 tablet_bootstrap.cc:492] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442: Bootstrap starting.
I20251025 20:02:40.580229 29770 raft_consensus.cc:740] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.582051 29770 consensus_queue.cc:260] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:40.582706 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f" is_pre_election: true
I20251025 20:02:40.586932 29772 ts_tablet_manager.cc:1434] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f: Time spent starting tablet: real 0.051s user 0.006s sys 0.005s
W20251025 20:02:40.586980 29618 leader_election.cc:343] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:40.589242 29772 tablet_bootstrap.cc:492] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f: Bootstrap starting.
I20251025 20:02:40.597064 29770 ts_tablet_manager.cc:1434] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32: Time spent starting tablet: real 0.038s user 0.010s sys 0.003s
I20251025 20:02:40.598626 29770 tablet_bootstrap.cc:492] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32: Bootstrap starting.
I20251025 20:02:40.623714 29771 tablet_bootstrap.cc:654] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.626547 29770 tablet_bootstrap.cc:654] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.630429 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
I20251025 20:02:40.630023 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c443311caa74ff4a1df53c5323f21b2" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
I20251025 20:02:40.632544 29653 raft_consensus.cc:2468] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 0.
I20251025 20:02:40.634068 29578 raft_consensus.cc:2468] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 0.
I20251025 20:02:40.637287 29542 leader_election.cc:304] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:40.637373 29617 leader_election.cc:304] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32; no voters: 10924e710604471a813b87ff5bbcfb8f
W20251025 20:02:40.642782 29679 tablet.cc:2378] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:02:40.643725 29776 raft_consensus.cc:2804] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:02:40.644150 29776 raft_consensus.cc:493] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:02:40.644435 29787 raft_consensus.cc:2804] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:02:40.645219 29787 raft_consensus.cc:493] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:02:40.645606 29787 raft_consensus.cc:3060] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:40.644712 29776 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:02:40.649509 29604 tablet.cc:2378] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:02:40.654875 29772 tablet_bootstrap.cc:654] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.662279 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c443311caa74ff4a1df53c5323f21b2" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f" is_pre_election: true
W20251025 20:02:40.664762 29543 leader_election.cc:343] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:40.666802 29778 raft_consensus.cc:493] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:40.667420 29778 raft_consensus.cc:515] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.671525 29776 raft_consensus.cc:515] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.668387 29787 raft_consensus.cc:515] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:40.674018 29778 leader_election.cc:290] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:40.684351 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:40.683736 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:40.679015 29776 leader_election.cc:290] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:40.688002 29578 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:40.693105 29770 tablet_bootstrap.cc:492] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32: No bootstrap required, opened a new log
I20251025 20:02:40.710594 29771 tablet_bootstrap.cc:492] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442: No bootstrap required, opened a new log
I20251025 20:02:40.690927 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c443311caa74ff4a1df53c5323f21b2" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
W20251025 20:02:40.690237 29618 leader_election.cc:343] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:40.689400 29787 leader_election.cc:290] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 election: Requested vote from peers 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:40.711195 29770 ts_tablet_manager.cc:1403] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32: Time spent bootstrapping tablet: real 0.113s user 0.029s sys 0.035s
I20251025 20:02:40.722489 29771 ts_tablet_manager.cc:1403] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442: Time spent bootstrapping tablet: real 0.143s user 0.020s sys 0.017s
I20251025 20:02:40.722036 29653 raft_consensus.cc:3060] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:40.734495 29771 raft_consensus.cc:359] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
W20251025 20:02:40.735255 29523 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:02:40.735687 29771 raft_consensus.cc:385] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.731516 29770 raft_consensus.cc:359] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.736003 29771 raft_consensus.cc:740] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.736950 29771 consensus_queue.cc:260] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.744236 29771 ts_tablet_manager.cc:1434] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442: Time spent starting tablet: real 0.020s user 0.001s sys 0.008s
I20251025 20:02:40.748152 29578 raft_consensus.cc:2468] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 1.
I20251025 20:02:40.748345 29772 tablet_bootstrap.cc:492] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f: No bootstrap required, opened a new log
I20251025 20:02:40.750372 29772 ts_tablet_manager.cc:1403] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f: Time spent bootstrapping tablet: real 0.161s user 0.015s sys 0.019s
I20251025 20:02:40.750980 29617 leader_election.cc:304] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32; no voters: 10924e710604471a813b87ff5bbcfb8f
I20251025 20:02:40.752535 29771 tablet_bootstrap.cc:492] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442: Bootstrap starting.
I20251025 20:02:40.752921 29776 raft_consensus.cc:2804] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:02:40.739845 29770 raft_consensus.cc:385] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.754004 29770 raft_consensus.cc:740] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.764629 29653 raft_consensus.cc:2468] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 1.
I20251025 20:02:40.769194 29542 leader_election.cc:304] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:40.772537 29787 raft_consensus.cc:2804] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:02:40.773206 29787 raft_consensus.cc:697] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Becoming Leader. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:40.773689 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c443311caa74ff4a1df53c5323f21b2" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:40.775818 29787 consensus_queue.cc:237] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:40.780117 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
I20251025 20:02:40.781437 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
I20251025 20:02:40.775391 29770 consensus_queue.cc:260] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
W20251025 20:02:40.784564 29691 leader_election.cc:343] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:40.775323 29776 raft_consensus.cc:697] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Becoming Leader. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:40.783056 29772 raft_consensus.cc:359] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:40.787771 29772 raft_consensus.cc:385] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.788111 29772 raft_consensus.cc:740] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Initialized, Role: FOLLOWER
W20251025 20:02:40.791841 29691 leader_election.cc:343] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:40.789978 29772 consensus_queue.cc:260] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:40.792649 29776 consensus_queue.cc:237] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.794492 29691 leader_election.cc:304] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f; no voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:40.797412 29727 raft_consensus.cc:3060] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:40.798447 29771 tablet_bootstrap.cc:654] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.799216 29778 raft_consensus.cc:2749] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:02:40.809574 29772 ts_tablet_manager.cc:1434] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f: Time spent starting tablet: real 0.055s user 0.005s sys 0.005s
I20251025 20:02:40.811017 29772 tablet_bootstrap.cc:492] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f: Bootstrap starting.
I20251025 20:02:40.813071 29770 ts_tablet_manager.cc:1434] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32: Time spent starting tablet: real 0.091s user 0.009s sys 0.012s
I20251025 20:02:40.815941 29770 tablet_bootstrap.cc:492] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32: Bootstrap starting.
I20251025 20:02:40.812177 29474 catalog_manager.cc:5649] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "23b530a2476d464eb636e2f3ea2e1442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: HEALTHY } } }
I20251025 20:02:40.830153 29772 tablet_bootstrap.cc:654] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.831326 29727 raft_consensus.cc:2468] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 1.
I20251025 20:02:40.839551 29473 catalog_manager.cc:5649] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 reported cstate change: term changed from 0 to 1, leader changed from <none> to e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194). New cstate: current_term: 1 leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:40.862718 29770 tablet_bootstrap.cc:654] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.863204 29771 tablet_bootstrap.cc:492] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442: No bootstrap required, opened a new log
I20251025 20:02:40.864862 29771 ts_tablet_manager.cc:1403] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442: Time spent bootstrapping tablet: real 0.113s user 0.021s sys 0.033s
I20251025 20:02:40.877727 29771 raft_consensus.cc:359] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.878649 29771 raft_consensus.cc:385] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.879125 29772 tablet_bootstrap.cc:492] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f: No bootstrap required, opened a new log
I20251025 20:02:40.878980 29771 raft_consensus.cc:740] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.879698 29772 ts_tablet_manager.cc:1403] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.031s
I20251025 20:02:40.880293 29771 consensus_queue.cc:260] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.885025 29772 raft_consensus.cc:359] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.886817 29771 ts_tablet_manager.cc:1434] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442: Time spent starting tablet: real 0.021s user 0.011s sys 0.001s
I20251025 20:02:40.889811 29772 raft_consensus.cc:385] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.890136 29772 raft_consensus.cc:740] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.890700 29772 consensus_queue.cc:260] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.891055 29771 tablet_bootstrap.cc:492] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442: Bootstrap starting.
I20251025 20:02:40.892620 29772 ts_tablet_manager.cc:1434] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f: Time spent starting tablet: real 0.013s user 0.004s sys 0.001s
I20251025 20:02:40.893492 29772 tablet_bootstrap.cc:492] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f: Bootstrap starting.
I20251025 20:02:40.889346 29787 raft_consensus.cc:493] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:40.894842 29787 raft_consensus.cc:515] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.898986 29787 leader_election.cc:290] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:40.898875 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
I20251025 20:02:40.900187 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f" is_pre_election: true
I20251025 20:02:40.900890 29727 raft_consensus.cc:2468] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 0.
I20251025 20:02:40.903348 29543 leader_election.cc:304] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:40.903785 29770 tablet_bootstrap.cc:492] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32: No bootstrap required, opened a new log
I20251025 20:02:40.904305 29787 raft_consensus.cc:2804] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:02:40.904341 29776 raft_consensus.cc:493] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:40.904839 29772 tablet_bootstrap.cc:654] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f: Neither blocks nor log segments found. Creating new log.
W20251025 20:02:40.905014 29542 leader_election.cc:343] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:40.904910 29776 raft_consensus.cc:515] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.907375 29776 leader_election.cc:290] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:40.908011 29787 raft_consensus.cc:493] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:02:40.908427 29787 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:40.910081 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
W20251025 20:02:40.911710 29617 leader_election.cc:343] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:40.913288 29770 ts_tablet_manager.cc:1403] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32: Time spent bootstrapping tablet: real 0.098s user 0.028s sys 0.012s
I20251025 20:02:40.912998 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f" is_pre_election: true
W20251025 20:02:40.915681 29618 leader_election.cc:343] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:40.916512 29771 tablet_bootstrap.cc:654] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.916162 29618 leader_election.cc:304] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e76383577c9d4685bf6d452bf2e7ec32; no voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442
I20251025 20:02:40.918186 29776 raft_consensus.cc:2749] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:02:40.918051 29770 raft_consensus.cc:359] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.918977 29770 raft_consensus.cc:385] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.919423 29770 raft_consensus.cc:740] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.919637 29787 raft_consensus.cc:515] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.920497 29770 consensus_queue.cc:260] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.923467 29787 leader_election.cc:290] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 election: Requested vote from peers e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:40.925674 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
W20251025 20:02:40.937217 29542 leader_election.cc:343] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:40.925683 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:40.938325 29727 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:40.924858 29770 ts_tablet_manager.cc:1434] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32: Time spent starting tablet: real 0.011s user 0.005s sys 0.002s
I20251025 20:02:40.946935 29770 tablet_bootstrap.cc:492] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32: Bootstrap starting.
I20251025 20:02:40.953256 29727 raft_consensus.cc:2468] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 1.
I20251025 20:02:40.954901 29543 leader_election.cc:304] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters: e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:40.956144 29787 raft_consensus.cc:2804] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:02:40.956768 29770 tablet_bootstrap.cc:654] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:40.956838 29787 raft_consensus.cc:697] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Becoming Leader. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:40.958253 29787 consensus_queue.cc:237] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.962214 29772 tablet_bootstrap.cc:492] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f: No bootstrap required, opened a new log
I20251025 20:02:40.962925 29772 ts_tablet_manager.cc:1403] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f: Time spent bootstrapping tablet: real 0.070s user 0.016s sys 0.007s
I20251025 20:02:40.964888 29771 tablet_bootstrap.cc:492] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442: No bootstrap required, opened a new log
I20251025 20:02:40.965513 29771 ts_tablet_manager.cc:1403] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442: Time spent bootstrapping tablet: real 0.075s user 0.030s sys 0.023s
I20251025 20:02:40.967556 29772 raft_consensus.cc:359] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.968818 29772 raft_consensus.cc:385] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.968950 29771 raft_consensus.cc:359] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.969645 29772 raft_consensus.cc:740] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.969921 29771 raft_consensus.cc:385] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:40.970425 29771 raft_consensus.cc:740] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Initialized, Role: FOLLOWER
I20251025 20:02:40.971350 29771 consensus_queue.cc:260] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.971026 29772 consensus_queue.cc:260] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.974436 29778 raft_consensus.cc:493] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:40.974983 29771 ts_tablet_manager.cc:1434] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20251025 20:02:40.975167 29778 raft_consensus.cc:515] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.976202 29771 tablet_bootstrap.cc:492] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442: Bootstrap starting.
I20251025 20:02:40.979390 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b067857e85d640b584839f045b12b019" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
I20251025 20:02:40.980717 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b067857e85d640b584839f045b12b019" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
W20251025 20:02:40.981722 29691 leader_election.cc:343] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:02:40.983493 29691 leader_election.cc:343] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:40.983932 29691 leader_election.cc:304] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f; no voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:40.985992 29770 tablet_bootstrap.cc:492] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32: No bootstrap required, opened a new log
I20251025 20:02:40.986950 29770 ts_tablet_manager.cc:1403] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.005s
I20251025 20:02:40.987102 29778 leader_election.cc:290] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:40.988082 29778 raft_consensus.cc:493] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:40.988600 29778 raft_consensus.cc:515] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:40.990892 29778 leader_election.cc:290] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:40.991449 29778 raft_consensus.cc:2749] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:02:40.994256 29772 ts_tablet_manager.cc:1434] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f: Time spent starting tablet: real 0.031s user 0.008s sys 0.003s
I20251025 20:02:40.995247 29770 raft_consensus.cc:359] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.000626 29770 raft_consensus.cc:385] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.001276 29770 raft_consensus.cc:740] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.003398 29770 consensus_queue.cc:260] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.007325 29770 ts_tablet_manager.cc:1434] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20251025 20:02:41.008533 29770 tablet_bootstrap.cc:492] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32: Bootstrap starting.
I20251025 20:02:41.011348 29772 tablet_bootstrap.cc:492] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f: Bootstrap starting.
I20251025 20:02:41.017560 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
I20251025 20:02:41.017714 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
I20251025 20:02:41.018482 29578 raft_consensus.cc:2468] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 0.
I20251025 20:02:41.018492 29653 raft_consensus.cc:2468] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 0.
I20251025 20:02:41.022087 29474 catalog_manager.cc:5649] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "23b530a2476d464eb636e2f3ea2e1442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:41.023723 29770 tablet_bootstrap.cc:654] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.025446 29691 leader_election.cc:304] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:41.026945 29814 raft_consensus.cc:2804] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:02:41.027562 29814 raft_consensus.cc:493] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:02:41.027998 29814 raft_consensus.cc:3060] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.028290 29776 raft_consensus.cc:493] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:41.028784 29776 raft_consensus.cc:515] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.045097 29771 tablet_bootstrap.cc:654] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.045481 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
I20251025 20:02:41.047127 29776 leader_election.cc:290] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:41.047389 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f" is_pre_election: true
I20251025 20:02:41.048360 29727 raft_consensus.cc:2468] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 0.
I20251025 20:02:41.050177 29772 tablet_bootstrap.cc:654] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.049940 29618 leader_election.cc:304] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:41.052377 29814 raft_consensus.cc:515] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
W20251025 20:02:41.052997 29617 leader_election.cc:343] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:41.054157 29776 raft_consensus.cc:2804] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:02:41.054656 29814 leader_election.cc:290] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:41.055505 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:41.055769 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:41.056142 29578 raft_consensus.cc:3060] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.056313 29653 raft_consensus.cc:3060] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.060981 29776 raft_consensus.cc:493] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:02:41.061647 29776 raft_consensus.cc:3060] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.064430 29653 raft_consensus.cc:2468] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 1.
I20251025 20:02:41.065748 29691 leader_election.cc:304] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:41.067819 29770 tablet_bootstrap.cc:492] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32: No bootstrap required, opened a new log
I20251025 20:02:41.068470 29770 ts_tablet_manager.cc:1403] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.007s
I20251025 20:02:41.072533 29770 raft_consensus.cc:359] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.073323 29770 raft_consensus.cc:385] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.073706 29770 raft_consensus.cc:740] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.074371 29814 raft_consensus.cc:2804] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:02:41.075853 29814 raft_consensus.cc:697] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Becoming Leader. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:41.076463 29776 raft_consensus.cc:515] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.077769 29814 consensus_queue.cc:237] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.080189 29577 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:41.080106 29770 consensus_queue.cc:260] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.081197 29578 raft_consensus.cc:2468] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 1.
W20251025 20:02:41.081925 29617 leader_election.cc:343] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:41.083057 29772 tablet_bootstrap.cc:492] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f: No bootstrap required, opened a new log
I20251025 20:02:41.083794 29772 ts_tablet_manager.cc:1403] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f: Time spent bootstrapping tablet: real 0.073s user 0.015s sys 0.014s
I20251025 20:02:41.085994 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:41.086668 29727 raft_consensus.cc:3060] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.096083 29727 raft_consensus.cc:2468] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 1.
I20251025 20:02:41.096835 29770 ts_tablet_manager.cc:1434] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32: Time spent starting tablet: real 0.028s user 0.008s sys 0.003s
I20251025 20:02:41.097447 29618 leader_election.cc:304] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32; no voters: 23b530a2476d464eb636e2f3ea2e1442
I20251025 20:02:41.098399 29770 tablet_bootstrap.cc:492] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32: Bootstrap starting.
I20251025 20:02:41.102056 29798 raft_consensus.cc:2804] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:02:41.102732 29798 raft_consensus.cc:697] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Becoming Leader. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:41.088985 29772 raft_consensus.cc:359] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.103412 29772 raft_consensus.cc:385] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.104071 29772 raft_consensus.cc:740] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.103705 29798 consensus_queue.cc:237] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.105517 29771 tablet_bootstrap.cc:492] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442: No bootstrap required, opened a new log
I20251025 20:02:41.106312 29771 ts_tablet_manager.cc:1403] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442: Time spent bootstrapping tablet: real 0.130s user 0.016s sys 0.017s
I20251025 20:02:41.110152 29771 raft_consensus.cc:359] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.111490 29771 raft_consensus.cc:385] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.105002 29772 consensus_queue.cc:260] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.114579 29772 ts_tablet_manager.cc:1434] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f: Time spent starting tablet: real 0.030s user 0.010s sys 0.000s
I20251025 20:02:41.090384 29776 leader_election.cc:290] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:41.116302 29772 tablet_bootstrap.cc:492] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f: Bootstrap starting.
I20251025 20:02:41.116700 29771 raft_consensus.cc:740] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.118811 29770 tablet_bootstrap.cc:654] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.123627 29771 consensus_queue.cc:260] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.130904 29771 ts_tablet_manager.cc:1434] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20251025 20:02:41.130554 29474 catalog_manager.cc:5649] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 reported cstate change: term changed from 0 to 1, leader changed from <none> to e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194). New cstate: current_term: 1 leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:41.132695 29771 tablet_bootstrap.cc:492] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442: Bootstrap starting.
I20251025 20:02:41.159360 29771 tablet_bootstrap.cc:654] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.168859 29472 catalog_manager.cc:5649] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10924e710604471a813b87ff5bbcfb8f (127.28.192.195). New cstate: current_term: 1 leader_uuid: "10924e710604471a813b87ff5bbcfb8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20251025 20:02:41.169595 29772 tablet_bootstrap.cc:654] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.176959 29770 tablet_bootstrap.cc:492] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32: No bootstrap required, opened a new log
I20251025 20:02:41.177654 29770 ts_tablet_manager.cc:1403] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32: Time spent bootstrapping tablet: real 0.080s user 0.020s sys 0.000s
I20251025 20:02:41.178301 29771 tablet_bootstrap.cc:492] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442: No bootstrap required, opened a new log
I20251025 20:02:41.181496 29771 ts_tablet_manager.cc:1403] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442: Time spent bootstrapping tablet: real 0.049s user 0.019s sys 0.006s
I20251025 20:02:41.183811 29770 raft_consensus.cc:359] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.184595 29770 raft_consensus.cc:385] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.184875 29770 raft_consensus.cc:740] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.185839 29771 raft_consensus.cc:359] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.185878 29770 consensus_queue.cc:260] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.187096 29771 raft_consensus.cc:385] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.189282 29770 ts_tablet_manager.cc:1434] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251025 20:02:41.190325 29770 tablet_bootstrap.cc:492] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32: Bootstrap starting.
I20251025 20:02:41.187563 29771 raft_consensus.cc:740] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.191726 29771 consensus_queue.cc:260] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.204046 29770 tablet_bootstrap.cc:654] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.195847 29771 ts_tablet_manager.cc:1434] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442: Time spent starting tablet: real 0.014s user 0.004s sys 0.006s
I20251025 20:02:41.205286 29772 tablet_bootstrap.cc:492] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f: No bootstrap required, opened a new log
I20251025 20:02:41.206115 29771 tablet_bootstrap.cc:492] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442: Bootstrap starting.
I20251025 20:02:41.206440 29772 ts_tablet_manager.cc:1403] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f: Time spent bootstrapping tablet: real 0.091s user 0.010s sys 0.012s
I20251025 20:02:41.212412 29772 raft_consensus.cc:359] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.213210 29772 raft_consensus.cc:385] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.213469 29772 raft_consensus.cc:740] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.214097 29772 consensus_queue.cc:260] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.217012 29772 ts_tablet_manager.cc:1434] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20251025 20:02:41.217955 29772 tablet_bootstrap.cc:492] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f: Bootstrap starting.
I20251025 20:02:41.226416 29771 tablet_bootstrap.cc:654] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.232127 29772 tablet_bootstrap.cc:654] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.232399 29787 consensus_queue.cc:1048] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.237016 29770 tablet_bootstrap.cc:492] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32: No bootstrap required, opened a new log
I20251025 20:02:41.237938 29770 ts_tablet_manager.cc:1403] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.011s
I20251025 20:02:41.240226 29814 raft_consensus.cc:493] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:41.240809 29814 raft_consensus.cc:515] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.241818 29771 tablet_bootstrap.cc:492] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442: No bootstrap required, opened a new log
I20251025 20:02:41.242334 29771 ts_tablet_manager.cc:1403] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.008s
I20251025 20:02:41.243091 29814 leader_election.cc:290] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:41.244983 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
I20251025 20:02:41.245708 29770 raft_consensus.cc:359] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.258373 29772 tablet_bootstrap.cc:492] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f: No bootstrap required, opened a new log
I20251025 20:02:41.247700 29771 raft_consensus.cc:359] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.245254 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
I20251025 20:02:41.268232 29772 ts_tablet_manager.cc:1403] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f: Time spent bootstrapping tablet: real 0.051s user 0.017s sys 0.017s
I20251025 20:02:41.268607 29653 raft_consensus.cc:2468] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 0.
I20251025 20:02:41.267338 29770 raft_consensus.cc:385] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 20:02:41.251533 29691 leader_election.cc:343] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:41.268265 29771 raft_consensus.cc:385] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.265079 29787 consensus_queue.cc:1048] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.270179 29771 raft_consensus.cc:740] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.270900 29691 leader_election.cc:304] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32; no voters: 23b530a2476d464eb636e2f3ea2e1442
I20251025 20:02:41.271919 29814 raft_consensus.cc:2804] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:02:41.272117 29770 raft_consensus.cc:740] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.272326 29814 raft_consensus.cc:493] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:02:41.272789 29814 raft_consensus.cc:3060] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.273190 29770 consensus_queue.cc:260] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.278811 29770 ts_tablet_manager.cc:1434] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20251025 20:02:41.270926 29771 consensus_queue.cc:260] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.282060 29771 ts_tablet_manager.cc:1434] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442: Time spent starting tablet: real 0.039s user 0.012s sys 0.008s
I20251025 20:02:41.283424 29771 tablet_bootstrap.cc:492] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442: Bootstrap starting.
I20251025 20:02:41.283320 29814 raft_consensus.cc:515] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.286159 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:41.287679 29578 raft_consensus.cc:3060] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.290198 29771 tablet_bootstrap.cc:654] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.296062 29578 raft_consensus.cc:2468] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 1.
I20251025 20:02:41.297505 29691 leader_election.cc:304] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:41.299041 29812 raft_consensus.cc:2804] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:02:41.300065 29772 raft_consensus.cc:359] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.299737 29812 raft_consensus.cc:697] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Becoming Leader. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:41.301457 29772 raft_consensus.cc:385] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.301877 29772 raft_consensus.cc:740] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.302073 29812 consensus_queue.cc:237] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.304008 29814 leader_election.cc:290] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:41.304694 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:41.304828 29772 consensus_queue.cc:260] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.305372 29652 raft_consensus.cc:3060] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.304025 29798 raft_consensus.cc:493] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:41.306421 29798 raft_consensus.cc:515] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.307210 29772 ts_tablet_manager.cc:1434] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f: Time spent starting tablet: real 0.039s user 0.001s sys 0.005s
I20251025 20:02:41.308210 29772 tablet_bootstrap.cc:492] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f: Bootstrap starting.
I20251025 20:02:41.308717 29798 leader_election.cc:290] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:41.311657 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a12402079514257b1bd8b657887b2bc" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f" is_pre_election: true
I20251025 20:02:41.311882 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a12402079514257b1bd8b657887b2bc" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
W20251025 20:02:41.314375 29618 leader_election.cc:343] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:41.319926 29772 tablet_bootstrap.cc:654] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f: Neither blocks nor log segments found. Creating new log.
I20251025 20:02:41.321924 29787 raft_consensus.cc:493] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:41.324344 29787 raft_consensus.cc:515] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.313242 29578 raft_consensus.cc:2468] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 0.
I20251025 20:02:41.321918 29798 raft_consensus.cc:493] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:41.327797 29798 raft_consensus.cc:515] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.328509 29727 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.330825 29798 leader_election.cc:290] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:41.333058 29577 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
I20251025 20:02:41.333946 29787 leader_election.cc:290] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:41.335629 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f" is_pre_election: true
I20251025 20:02:41.331653 29798 consensus_queue.cc:1048] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251025 20:02:41.337754 29725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b067857e85d640b584839f045b12b019" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f" is_pre_election: true
I20251025 20:02:41.338832 29725 raft_consensus.cc:2468] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 0.
I20251025 20:02:41.341285 29651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b067857e85d640b584839f045b12b019" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
I20251025 20:02:41.342075 29651 raft_consensus.cc:2468] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 0.
I20251025 20:02:41.337200 29726 raft_consensus.cc:2393] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e76383577c9d4685bf6d452bf2e7ec32 in current term 1: Already voted for candidate 23b530a2476d464eb636e2f3ea2e1442 in this term.
I20251025 20:02:41.345729 29617 leader_election.cc:304] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32; no voters: 10924e710604471a813b87ff5bbcfb8f
I20251025 20:02:41.348024 29542 leader_election.cc:304] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:41.353277 29787 raft_consensus.cc:2804] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:02:41.353753 29787 raft_consensus.cc:493] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:02:41.354280 29787 raft_consensus.cc:3060] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.354398 29798 raft_consensus.cc:2804] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:02:41.354835 29798 raft_consensus.cc:493] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:02:41.355248 29798 raft_consensus.cc:3060] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.355679 29618 leader_election.cc:304] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e76383577c9d4685bf6d452bf2e7ec32; no voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442
I20251025 20:02:41.357645 29776 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.361143 29814 raft_consensus.cc:493] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:41.359128 29652 raft_consensus.cc:2468] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 1.
I20251025 20:02:41.366616 29814 raft_consensus.cc:515] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.369539 29776 raft_consensus.cc:2749] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:02:41.379685 29771 tablet_bootstrap.cc:492] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442: No bootstrap required, opened a new log
I20251025 20:02:41.380280 29771 ts_tablet_manager.cc:1403] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442: Time spent bootstrapping tablet: real 0.097s user 0.013s sys 0.012s
I20251025 20:02:41.376405 29776 consensus_queue.cc:1048] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251025 20:02:41.380903 29787 raft_consensus.cc:515] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.384083 29771 raft_consensus.cc:359] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.385041 29771 raft_consensus.cc:385] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.384752 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b067857e85d640b584839f045b12b019" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:41.385622 29771 raft_consensus.cc:740] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.386305 29653 raft_consensus.cc:3060] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.387811 29771 consensus_queue.cc:260] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.389065 29812 raft_consensus.cc:493] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:41.390805 29787 leader_election.cc:290] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 election: Requested vote from peers e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:41.394685 29771 ts_tablet_manager.cc:1434] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s
I20251025 20:02:41.396066 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b067857e85d640b584839f045b12b019" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:41.401080 29726 raft_consensus.cc:3060] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.392210 29812 raft_consensus.cc:515] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.410633 29812 leader_election.cc:290] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:41.411919 29577 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
I20251025 20:02:41.412055 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
I20251025 20:02:41.412767 29577 raft_consensus.cc:2468] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 0.
I20251025 20:02:41.412771 29652 raft_consensus.cc:2468] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 0.
I20251025 20:02:41.417099 29691 leader_election.cc:304] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:41.416807 29798 raft_consensus.cc:515] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.422523 29725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a12402079514257b1bd8b657887b2bc" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:41.422608 29577 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a12402079514257b1bd8b657887b2bc" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:41.423238 29812 raft_consensus.cc:2804] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:02:41.423383 29577 raft_consensus.cc:3060] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.423678 29812 raft_consensus.cc:493] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:02:41.430019 29812 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:02:41.425945 29618 leader_election.cc:343] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:02:41.424651 29798 leader_election.cc:290] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:41.441601 29772 tablet_bootstrap.cc:492] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f: No bootstrap required, opened a new log
I20251025 20:02:41.442286 29772 ts_tablet_manager.cc:1403] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f: Time spent bootstrapping tablet: real 0.134s user 0.015s sys 0.020s
I20251025 20:02:41.443795 29726 raft_consensus.cc:2468] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 1.
I20251025 20:02:41.445035 29653 raft_consensus.cc:2468] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 1.
I20251025 20:02:41.445149 29543 leader_election.cc:304] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:41.447331 29787 raft_consensus.cc:2804] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:02:41.447839 29787 raft_consensus.cc:697] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Becoming Leader. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:41.448851 29787 consensus_queue.cc:237] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.449234 29812 raft_consensus.cc:515] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.449753 29772 raft_consensus.cc:359] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.450448 29772 raft_consensus.cc:385] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:02:41.450727 29772 raft_consensus.cc:740] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Initialized, Role: FOLLOWER
I20251025 20:02:41.452530 29772 consensus_queue.cc:260] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.452232 29812 leader_election.cc:290] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:41.454408 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:41.459017 29772 ts_tablet_manager.cc:1434] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f: Time spent starting tablet: real 0.016s user 0.008s sys 0.001s
I20251025 20:02:41.457049 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:41.456768 29578 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.459848 29653 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.475596 29577 raft_consensus.cc:2468] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 1.
I20251025 20:02:41.477792 29653 raft_consensus.cc:2468] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 1.
I20251025 20:02:41.478255 29617 leader_election.cc:304] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32; no voters: 10924e710604471a813b87ff5bbcfb8f
I20251025 20:02:41.479182 29691 leader_election.cc:304] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:41.479529 29798 raft_consensus.cc:2804] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:02:41.478222 29578 raft_consensus.cc:2468] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 1.
I20251025 20:02:41.480589 29836 raft_consensus.cc:2804] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:02:41.481932 29812 raft_consensus.cc:493] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:41.482409 29836 raft_consensus.cc:697] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Becoming Leader. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:41.482440 29812 raft_consensus.cc:515] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.483727 29836 consensus_queue.cc:237] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.486876 29475 catalog_manager.cc:5649] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10924e710604471a813b87ff5bbcfb8f (127.28.192.195). New cstate: current_term: 1 leader_uuid: "10924e710604471a813b87ff5bbcfb8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20251025 20:02:41.488910 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
I20251025 20:02:41.489566 29812 leader_election.cc:290] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:41.483062 29798 raft_consensus.cc:697] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Becoming Leader. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:41.494042 29798 consensus_queue.cc:237] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.497404 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a12402079514257b1bd8b657887b2bc" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
I20251025 20:02:41.484100 29814 leader_election.cc:290] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:41.501962 29653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
I20251025 20:02:41.503602 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a12402079514257b1bd8b657887b2bc" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b530a2476d464eb636e2f3ea2e1442" is_pre_election: true
I20251025 20:02:41.503786 29691 leader_election.cc:304] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f; no voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:41.504511 29578 raft_consensus.cc:2393] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10924e710604471a813b87ff5bbcfb8f in current term 1: Already voted for candidate e76383577c9d4685bf6d452bf2e7ec32 in this term.
I20251025 20:02:41.504940 29814 raft_consensus.cc:2749] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:02:41.506460 29691 leader_election.cc:304] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f; no voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:41.508740 29814 raft_consensus.cc:3060] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.519613 29787 consensus_queue.cc:1048] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.519496 29472 catalog_manager.cc:5649] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "23b530a2476d464eb636e2f3ea2e1442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:41.529783 29814 raft_consensus.cc:2749] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:02:41.535401 29787 consensus_queue.cc:1048] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.536028 29475 catalog_manager.cc:5649] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 reported cstate change: term changed from 0 to 1, leader changed from <none> to e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194). New cstate: current_term: 1 leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:41.537321 29787 raft_consensus.cc:493] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:02:41.537828 29787 raft_consensus.cc:515] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.540310 29787 leader_election.cc:290] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:41.542027 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10924e710604471a813b87ff5bbcfb8f" is_pre_election: true
I20251025 20:02:41.543042 29727 raft_consensus.cc:2393] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23b530a2476d464eb636e2f3ea2e1442 in current term 1: Already voted for candidate e76383577c9d4685bf6d452bf2e7ec32 in this term.
I20251025 20:02:41.544252 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32" is_pre_election: true
I20251025 20:02:41.550297 29543 leader_election.cc:304] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442; no voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:41.554543 29814 consensus_queue.cc:1048] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.555913 29798 consensus_queue.cc:1048] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:02:41.557984 29474 catalog_manager.cc:5649] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10924e710604471a813b87ff5bbcfb8f (127.28.192.195). New cstate: current_term: 1 leader_uuid: "10924e710604471a813b87ff5bbcfb8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20251025 20:02:41.568321 29787 raft_consensus.cc:3060] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:02:41.596613 29787 raft_consensus.cc:2749] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:02:41.614492 29778 consensus_queue.cc:1048] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.619367 29797 consensus_queue.cc:1048] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.720618 29727 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0"
dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
mode: GRACEFUL
new_leader_uuid: "23b530a2476d464eb636e2f3ea2e1442"
from {username='slave'} at 127.0.0.1:60934
I20251025 20:02:41.721304 29727 raft_consensus.cc:606] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Received request to transfer leadership to TS 23b530a2476d464eb636e2f3ea2e1442
I20251025 20:02:41.742745 29653 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
mode: GRACEFUL
new_leader_uuid: "23b530a2476d464eb636e2f3ea2e1442"
from {username='slave'} at 127.0.0.1:35696
I20251025 20:02:41.743570 29653 raft_consensus.cc:606] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Received request to transfer leadership to TS 23b530a2476d464eb636e2f3ea2e1442
I20251025 20:02:41.752364 29727 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38"
dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
mode: GRACEFUL
new_leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:60934
I20251025 20:02:41.753165 29727 raft_consensus.cc:606] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Received request to transfer leadership to TS e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:41.767529 29727 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea"
dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
mode: GRACEFUL
new_leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:60934
I20251025 20:02:41.768512 29727 raft_consensus.cc:606] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Received request to transfer leadership to TS e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:41.789145 29578 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b067857e85d640b584839f045b12b019"
dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
mode: GRACEFUL
new_leader_uuid: "10924e710604471a813b87ff5bbcfb8f"
from {username='slave'} at 127.0.0.1:50030
I20251025 20:02:41.791002 29578 raft_consensus.cc:606] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Received request to transfer leadership to TS 10924e710604471a813b87ff5bbcfb8f
I20251025 20:02:41.790866 29812 consensus_queue.cc:1048] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.809832 29836 consensus_queue.cc:1048] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.823053 29812 raft_consensus.cc:993] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f: : Instructing follower e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:41.823642 29836 raft_consensus.cc:1081] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Signalling peer e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:41.825629 29653 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:35682
I20251025 20:02:41.826406 29653 raft_consensus.cc:493] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:41.826792 29653 raft_consensus.cc:3060] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:41.835700 29653 raft_consensus.cc:515] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.838001 29653 leader_election.cc:290] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 2 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:41.839432 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:41.840181 29727 raft_consensus.cc:3055] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:02:41.839957 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:41.840561 29727 raft_consensus.cc:740] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:41.841573 29578 raft_consensus.cc:3060] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:41.842506 29727 consensus_queue.cc:260] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.844260 29727 raft_consensus.cc:3060] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:41.850915 29578 raft_consensus.cc:2468] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 2.
I20251025 20:02:41.852713 29617 leader_election.cc:304] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:41.853404 29727 raft_consensus.cc:2468] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 2.
I20251025 20:02:41.853694 29797 raft_consensus.cc:2804] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:02:41.854239 29797 raft_consensus.cc:697] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Becoming Leader. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:41.856184 29797 consensus_queue.cc:237] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.866809 29474 catalog_manager.cc:5649] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 reported cstate change: term changed from 1 to 2, leader changed from 10924e710604471a813b87ff5bbcfb8f (127.28.192.195) to e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194). New cstate: current_term: 2 leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:41.900629 29787 consensus_queue.cc:1048] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.902215 29836 consensus_queue.cc:1048] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:02:41.911536 29787 raft_consensus.cc:993] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442: : Instructing follower 10924e710604471a813b87ff5bbcfb8f to start an election
I20251025 20:02:41.912036 29787 raft_consensus.cc:1081] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Signalling peer 10924e710604471a813b87ff5bbcfb8f to start an election
I20251025 20:02:41.918493 29726 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b067857e85d640b584839f045b12b019"
dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
from {username='slave'} at 127.0.0.1:60922
I20251025 20:02:41.919164 29726 raft_consensus.cc:493] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:41.919580 29726 raft_consensus.cc:3060] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:41.930299 29726 raft_consensus.cc:515] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.931039 29836 consensus_queue.cc:1048] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:02:41.937682 29787 consensus_queue.cc:1048] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:02:41.953073 29726 leader_election.cc:290] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 2 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:41.955585 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b067857e85d640b584839f045b12b019" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:41.955976 29651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b067857e85d640b584839f045b12b019" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:41.956512 29578 raft_consensus.cc:3055] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:02:41.956861 29578 raft_consensus.cc:740] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:41.957783 29578 consensus_queue.cc:260] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.963012 29578 raft_consensus.cc:3060] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:41.974350 29578 raft_consensus.cc:2468] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 2.
I20251025 20:02:41.975648 29691 leader_election.cc:304] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters: e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:41.976588 29836 raft_consensus.cc:2804] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:02:41.977078 29836 raft_consensus.cc:697] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Becoming Leader. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:41.978281 29836 consensus_queue.cc:237] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:41.989414 29778 raft_consensus.cc:993] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f: : Instructing follower 23b530a2476d464eb636e2f3ea2e1442 to start an election
I20251025 20:02:41.989866 29778 raft_consensus.cc:1081] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Signalling peer 23b530a2476d464eb636e2f3ea2e1442 to start an election
I20251025 20:02:41.991535 29577 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0"
dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
from {username='slave'} at 127.0.0.1:50022
I20251025 20:02:41.991044 29474 catalog_manager.cc:5649] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f reported cstate change: term changed from 1 to 2, leader changed from 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193) to 10924e710604471a813b87ff5bbcfb8f (127.28.192.195). New cstate: current_term: 2 leader_uuid: "10924e710604471a813b87ff5bbcfb8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20251025 20:02:41.992034 29577 raft_consensus.cc:493] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:41.993312 29577 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:42.001199 29577 raft_consensus.cc:515] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:42.004890 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:42.005455 29652 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:42.005328 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:42.006072 29726 raft_consensus.cc:3055] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:02:42.006453 29726 raft_consensus.cc:740] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:42.007359 29726 consensus_queue.cc:260] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:42.008575 29726 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:42.004715 29797 consensus_queue.cc:1048] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:02:42.015503 29776 raft_consensus.cc:993] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32: : Instructing follower 23b530a2476d464eb636e2f3ea2e1442 to start an election
I20251025 20:02:42.016031 29798 raft_consensus.cc:1081] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Signalling peer 23b530a2476d464eb636e2f3ea2e1442 to start an election
I20251025 20:02:42.016914 29798 consensus_queue.cc:1048] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:02:42.017799 29726 raft_consensus.cc:2468] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 2.
I20251025 20:02:42.018023 29576 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99"
dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
from {username='slave'} at 127.0.0.1:50006
I20251025 20:02:42.018606 29576 raft_consensus.cc:493] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:42.018990 29576 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:42.019106 29543 leader_election.cc:304] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:42.020898 29787 raft_consensus.cc:2804] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:02:42.021102 29652 raft_consensus.cc:2468] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 2.
I20251025 20:02:42.021457 29787 raft_consensus.cc:697] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 LEADER]: Becoming Leader. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:42.022469 29787 consensus_queue.cc:237] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:42.004220 29577 leader_election.cc:290] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 2 election: Requested vote from peers e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:42.031925 29576 raft_consensus.cc:515] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:42.034862 29576 leader_election.cc:290] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 2 election: Requested vote from peers e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:42.038398 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:42.039095 29726 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:42.048910 29726 raft_consensus.cc:2468] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 2.
I20251025 20:02:42.051440 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:42.051712 29543 leader_election.cc:304] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:42.052152 29652 raft_consensus.cc:3055] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:02:42.052525 29652 raft_consensus.cc:740] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:42.053017 29787 raft_consensus.cc:2804] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:02:42.053495 29787 raft_consensus.cc:697] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 LEADER]: Becoming Leader. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:42.054414 29787 consensus_queue.cc:237] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:42.049126 29475 catalog_manager.cc:5649] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 reported cstate change: term changed from 1 to 2, leader changed from 10924e710604471a813b87ff5bbcfb8f (127.28.192.195) to 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "23b530a2476d464eb636e2f3ea2e1442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:42.053471 29652 consensus_queue.cc:260] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:42.061012 29652 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:42.069236 29652 raft_consensus.cc:2468] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 2.
I20251025 20:02:42.082870 29475 catalog_manager.cc:5649] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 reported cstate change: term changed from 1 to 2, leader changed from e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194) to 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "23b530a2476d464eb636e2f3ea2e1442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:42.085512 29814 raft_consensus.cc:993] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f: : Instructing follower e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:42.086019 29836 raft_consensus.cc:1081] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Signalling peer e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:42.088085 29652 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:35682
I20251025 20:02:42.088833 29652 raft_consensus.cc:493] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:42.089224 29652 raft_consensus.cc:3060] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:42.097064 29652 raft_consensus.cc:515] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:42.099368 29652 leader_election.cc:290] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 2 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:42.099980 29576 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:42.100085 29727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:42.100744 29576 raft_consensus.cc:3060] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:42.100903 29727 raft_consensus.cc:3055] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:02:42.101323 29727 raft_consensus.cc:740] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:42.102299 29727 consensus_queue.cc:260] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:42.103693 29727 raft_consensus.cc:3060] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:42.106415 29576 raft_consensus.cc:2468] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 2.
I20251025 20:02:42.108759 29617 leader_election.cc:304] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:42.109531 29798 raft_consensus.cc:2804] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:02:42.110011 29798 raft_consensus.cc:697] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Becoming Leader. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:42.110884 29798 consensus_queue.cc:237] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:42.112008 29727 raft_consensus.cc:2468] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 2.
I20251025 20:02:42.121658 29475 catalog_manager.cc:5649] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 reported cstate change: term changed from 1 to 2, leader changed from 10924e710604471a813b87ff5bbcfb8f (127.28.192.195) to e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194). New cstate: current_term: 2 leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:42.302052 29727 raft_consensus.cc:1275] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:42.303678 29798 consensus_queue.cc:1048] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:42.314093 29576 raft_consensus.cc:1275] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:42.315897 29798 consensus_queue.cc:1048] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:42.404956 29576 raft_consensus.cc:1275] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Refusing update from remote peer 10924e710604471a813b87ff5bbcfb8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:42.406567 29836 consensus_queue.cc:1048] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:42.414992 29652 raft_consensus.cc:3060] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:42.425105 29652 raft_consensus.cc:1275] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Refusing update from remote peer 10924e710604471a813b87ff5bbcfb8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:42.426779 29812 consensus_queue.cc:1048] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:42.541019 29726 raft_consensus.cc:1275] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Refusing update from remote peer 23b530a2476d464eb636e2f3ea2e1442: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:42.542477 29787 consensus_queue.cc:1048] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:42.554060 29652 raft_consensus.cc:1275] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Refusing update from remote peer 23b530a2476d464eb636e2f3ea2e1442: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:42.557793 29777 consensus_queue.cc:1048] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:02:42.593335 29726 raft_consensus.cc:1275] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Refusing update from remote peer 23b530a2476d464eb636e2f3ea2e1442: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:42.595029 29777 consensus_queue.cc:1048] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:02:42.605068 29652 raft_consensus.cc:1275] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Refusing update from remote peer 23b530a2476d464eb636e2f3ea2e1442: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:42.607731 29845 consensus_queue.cc:1048] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:02:42.646708 29726 raft_consensus.cc:1275] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:42.648240 29798 consensus_queue.cc:1048] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:02:42.658609 29576 raft_consensus.cc:1275] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:42.660432 29776 consensus_queue.cc:1048] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:44.822914 29443 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20251025 20:02:44.823796 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:44.846213 29652 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
mode: GRACEFUL
new_leader_uuid: "10924e710604471a813b87ff5bbcfb8f"
from {username='slave'} at 127.0.0.1:35700
I20251025 20:02:44.846824 29652 raft_consensus.cc:606] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Received request to transfer leadership to TS 10924e710604471a813b87ff5bbcfb8f
I20251025 20:02:44.858314 29576 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284"
dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
mode: GRACEFUL
new_leader_uuid: "10924e710604471a813b87ff5bbcfb8f"
from {username='slave'} at 127.0.0.1:50042
I20251025 20:02:44.858886 29576 raft_consensus.cc:606] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Received request to transfer leadership to TS 10924e710604471a813b87ff5bbcfb8f
I20251025 20:02:44.860023 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251025 20:02:44.860492 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:44.916132 29868 raft_consensus.cc:993] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32: : Instructing follower 10924e710604471a813b87ff5bbcfb8f to start an election
I20251025 20:02:44.916687 29873 raft_consensus.cc:1081] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Signalling peer 10924e710604471a813b87ff5bbcfb8f to start an election
I20251025 20:02:44.918264 29727 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6"
dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
from {username='slave'} at 127.0.0.1:60918
I20251025 20:02:44.918751 29727 raft_consensus.cc:493] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:44.919005 29727 raft_consensus.cc:3060] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:44.924223 29727 raft_consensus.cc:515] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:44.926173 29727 leader_election.cc:290] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 2 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:44.927382 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:44.927228 29576 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:44.927929 29652 raft_consensus.cc:3055] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:02:44.927997 29576 raft_consensus.cc:3060] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:44.928369 29652 raft_consensus.cc:740] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:44.929235 29652 consensus_queue.cc:260] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:44.930320 29652 raft_consensus.cc:3060] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:44.933804 29576 raft_consensus.cc:2468] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 2.
I20251025 20:02:44.935108 29691 leader_election.cc:304] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:44.935701 29652 raft_consensus.cc:2468] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 2.
I20251025 20:02:44.935927 29812 raft_consensus.cc:2804] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:02:44.937395 29812 raft_consensus.cc:697] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Becoming Leader. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:44.938295 29812 consensus_queue.cc:237] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:44.945957 29472 catalog_manager.cc:5649] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f reported cstate change: term changed from 1 to 2, leader changed from e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194) to 10924e710604471a813b87ff5bbcfb8f (127.28.192.195). New cstate: current_term: 2 leader_uuid: "10924e710604471a813b87ff5bbcfb8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20251025 20:02:45.320246 29787 raft_consensus.cc:993] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442: : Instructing follower 10924e710604471a813b87ff5bbcfb8f to start an election
I20251025 20:02:45.320736 29869 raft_consensus.cc:1081] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Signalling peer 10924e710604471a813b87ff5bbcfb8f to start an election
I20251025 20:02:45.322342 29727 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284"
dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
from {username='slave'} at 127.0.0.1:60922
I20251025 20:02:45.322872 29727 raft_consensus.cc:493] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:45.323185 29727 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:45.328490 29727 raft_consensus.cc:515] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:45.330319 29727 leader_election.cc:290] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 2 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:45.331146 29576 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:45.331410 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:45.332043 29652 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:45.332571 29576 raft_consensus.cc:3055] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:02:45.332865 29576 raft_consensus.cc:740] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:45.333715 29576 consensus_queue.cc:260] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:45.334843 29576 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:45.338523 29652 raft_consensus.cc:2468] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 2.
I20251025 20:02:45.339780 29691 leader_election.cc:304] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:45.340449 29576 raft_consensus.cc:2468] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 2.
I20251025 20:02:45.340695 29812 raft_consensus.cc:2804] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:02:45.341167 29812 raft_consensus.cc:697] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Becoming Leader. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:45.342070 29812 consensus_queue.cc:237] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:45.350025 29472 catalog_manager.cc:5649] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f reported cstate change: term changed from 1 to 2, leader changed from 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193) to 10924e710604471a813b87ff5bbcfb8f (127.28.192.195). New cstate: current_term: 2 leader_uuid: "10924e710604471a813b87ff5bbcfb8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20251025 20:02:45.502035 29576 raft_consensus.cc:1275] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Refusing update from remote peer 10924e710604471a813b87ff5bbcfb8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:45.503357 29812 consensus_queue.cc:1048] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:45.516316 29652 raft_consensus.cc:1275] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Refusing update from remote peer 10924e710604471a813b87ff5bbcfb8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:45.518417 29874 consensus_queue.cc:1048] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:45.802225 29652 raft_consensus.cc:1275] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Refusing update from remote peer 10924e710604471a813b87ff5bbcfb8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:45.803515 29883 consensus_queue.cc:1048] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:45.813843 29576 raft_consensus.cc:1275] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Refusing update from remote peer 10924e710604471a813b87ff5bbcfb8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:45.816453 29812 consensus_queue.cc:1048] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251025 20:02:45.861410 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:45.866623 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:02:45.867100 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:46.868110 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:46.873368 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:02:46.873829 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:47.875351 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:47.880932 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:02:47.881417 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:48.882550 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:48.887414 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:02:48.887843 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:49.888824 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:49.893865 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:02:49.894402 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:50.895507 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:50.900556 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:02:50.901048 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:51.902154 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:51.908025 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:02:51.908603 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:52.909583 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:52.914661 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:02:52.915159 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:53.916175 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:53.921142 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:02:53.921654 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:54.949165 29443 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20251025 20:02:54.953774 29578 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7c443311caa74ff4a1df53c5323f21b2"
dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
mode: GRACEFUL
new_leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:50030
I20251025 20:02:54.954555 29578 raft_consensus.cc:606] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Received request to transfer leadership to TS e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:54.959757 29726 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284"
dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
mode: GRACEFUL
new_leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:60934
I20251025 20:02:54.960279 29726 raft_consensus.cc:606] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Received request to transfer leadership to TS e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:54.965579 29578 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0"
dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
mode: GRACEFUL
new_leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:50030
I20251025 20:02:54.966192 29578 raft_consensus.cc:606] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 LEADER]: Received request to transfer leadership to TS e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:54.970798 29578 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99"
dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
mode: GRACEFUL
new_leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:50030
I20251025 20:02:54.971333 29578 raft_consensus.cc:606] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 LEADER]: Received request to transfer leadership to TS e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:54.980439 29726 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6"
dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
mode: GRACEFUL
new_leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:60934
I20251025 20:02:54.980964 29726 raft_consensus.cc:606] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Received request to transfer leadership to TS e76383577c9d4685bf6d452bf2e7ec32
I20251025 20:02:55.048254 29894 raft_consensus.cc:993] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442: : Instructing follower e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:55.048795 29895 raft_consensus.cc:1081] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 LEADER]: Signalling peer e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:55.050418 29652 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:35678
I20251025 20:02:55.050910 29652 raft_consensus.cc:493] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:55.051225 29652 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.056727 29652 raft_consensus.cc:515] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.058467 29652 leader_election.cc:290] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 3 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:55.059610 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:55.059610 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:55.060520 29726 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.060590 29578 raft_consensus.cc:3055] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:02:55.060979 29578 raft_consensus.cc:740] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:55.061924 29578 consensus_queue.cc:260] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.063406 29578 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.068611 29726 raft_consensus.cc:2468] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 3.
I20251025 20:02:55.070318 29618 leader_election.cc:304] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:55.070989 29578 raft_consensus.cc:2468] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 3.
I20251025 20:02:55.071271 29893 raft_consensus.cc:2804] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:02:55.072302 29893 raft_consensus.cc:697] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Becoming Leader. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:55.073365 29893 consensus_queue.cc:237] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.081459 29472 catalog_manager.cc:5649] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 reported cstate change: term changed from 2 to 3, leader changed from 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193) to e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194). New cstate: current_term: 3 leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:55.115693 29902 raft_consensus.cc:993] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f: : Instructing follower e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:55.116284 29891 raft_consensus.cc:1081] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Signalling peer e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:55.117669 29652 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:35682
I20251025 20:02:55.118152 29652 raft_consensus.cc:493] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:55.118446 29652 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.122932 29652 raft_consensus.cc:515] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.124357 29652 leader_election.cc:290] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 3 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:55.125332 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:55.125454 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:55.125968 29726 raft_consensus.cc:3055] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:02:55.125938 29578 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.126356 29726 raft_consensus.cc:740] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:55.126997 29726 consensus_queue.cc:260] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.127907 29726 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.131471 29578 raft_consensus.cc:2468] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 3.
I20251025 20:02:55.132272 29726 raft_consensus.cc:2468] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 3.
I20251025 20:02:55.132845 29617 leader_election.cc:304] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b530a2476d464eb636e2f3ea2e1442, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:55.133673 29893 raft_consensus.cc:2804] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:02:55.134186 29893 raft_consensus.cc:697] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Becoming Leader. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:55.135089 29893 consensus_queue.cc:237] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.142251 29472 catalog_manager.cc:5649] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 reported cstate change: term changed from 2 to 3, leader changed from 10924e710604471a813b87ff5bbcfb8f (127.28.192.195) to e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194). New cstate: current_term: 3 leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:55.190142 29894 raft_consensus.cc:993] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442: : Instructing follower e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:55.190622 29895 raft_consensus.cc:1081] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 LEADER]: Signalling peer e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:55.192150 29652 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:35678
I20251025 20:02:55.192647 29652 raft_consensus.cc:493] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:55.192912 29652 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.197575 29652 raft_consensus.cc:515] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.199011 29652 leader_election.cc:290] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 3 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:55.200094 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:55.200001 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:55.200680 29726 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.200763 29578 raft_consensus.cc:3055] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:02:55.201190 29578 raft_consensus.cc:740] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:55.201885 29578 consensus_queue.cc:260] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.202908 29578 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.205358 29726 raft_consensus.cc:2468] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 3.
I20251025 20:02:55.206327 29618 leader_election.cc:304] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:55.207029 29893 raft_consensus.cc:2804] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:02:55.207576 29893 raft_consensus.cc:697] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Becoming Leader. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:55.208889 29578 raft_consensus.cc:2468] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 3.
I20251025 20:02:55.208654 29893 consensus_queue.cc:237] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.216318 29902 raft_consensus.cc:993] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f: : Instructing follower e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:55.216775 29891 raft_consensus.cc:1081] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Signalling peer e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:55.216976 29472 catalog_manager.cc:5649] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 reported cstate change: term changed from 2 to 3, leader changed from 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193) to e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194). New cstate: current_term: 3 leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:55.218094 29652 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:35682
I20251025 20:02:55.218529 29652 raft_consensus.cc:493] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:55.218782 29652 raft_consensus.cc:3060] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.224570 29652 raft_consensus.cc:515] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.226570 29652 leader_election.cc:290] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 3 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:55.227928 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:55.228617 29726 raft_consensus.cc:3055] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:02:55.228952 29726 raft_consensus.cc:740] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:55.229501 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ae63ee9cca14dae8be38ff2b05c01a6" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:55.229727 29726 consensus_queue.cc:260] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.230165 29578 raft_consensus.cc:3060] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.231050 29726 raft_consensus.cc:3060] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:55.235918 29726 raft_consensus.cc:2468] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 3.
I20251025 20:02:55.236099 29578 raft_consensus.cc:2468] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 3.
I20251025 20:02:55.236925 29618 leader_election.cc:304] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:55.237744 29893 raft_consensus.cc:2804] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:02:55.238205 29893 raft_consensus.cc:697] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Becoming Leader. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:55.239040 29893 consensus_queue.cc:237] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:55.246109 29472 catalog_manager.cc:5649] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 reported cstate change: term changed from 2 to 3, leader changed from 10924e710604471a813b87ff5bbcfb8f (127.28.192.195) to e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194). New cstate: current_term: 3 leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:55.313522 29894 raft_consensus.cc:993] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442: : Instructing follower e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:55.314031 29895 raft_consensus.cc:1081] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Signalling peer e76383577c9d4685bf6d452bf2e7ec32 to start an election
I20251025 20:02:55.315409 29652 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "7c443311caa74ff4a1df53c5323f21b2"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
from {username='slave'} at 127.0.0.1:35678
I20251025 20:02:55.315837 29652 raft_consensus.cc:493] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:55.316170 29652 raft_consensus.cc:3060] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:55.321556 29652 raft_consensus.cc:515] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:55.323133 29652 leader_election.cc:290] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 2 election: Requested vote from peers 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807), 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283)
I20251025 20:02:55.324079 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c443311caa74ff4a1df53c5323f21b2" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:55.324294 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c443311caa74ff4a1df53c5323f21b2" candidate_uuid: "e76383577c9d4685bf6d452bf2e7ec32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:55.324707 29726 raft_consensus.cc:3060] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:55.324824 29578 raft_consensus.cc:3055] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:02:55.325176 29578 raft_consensus.cc:740] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:55.326087 29578 consensus_queue.cc:260] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:55.327215 29578 raft_consensus.cc:3060] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:02:55.331611 29726 raft_consensus.cc:2468] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 2.
I20251025 20:02:55.332772 29618 leader_election.cc:304] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, e76383577c9d4685bf6d452bf2e7ec32; no voters:
I20251025 20:02:55.333561 29893 raft_consensus.cc:2804] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:02:55.334161 29893 raft_consensus.cc:697] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Becoming Leader. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:55.335661 29893 consensus_queue.cc:237] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } }
I20251025 20:02:55.336282 29578 raft_consensus.cc:2468] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76383577c9d4685bf6d452bf2e7ec32 in term 2.
I20251025 20:02:55.344379 29472 catalog_manager.cc:5649] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 reported cstate change: term changed from 1 to 2, leader changed from 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193) to e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194). New cstate: current_term: 2 leader_uuid: "e76383577c9d4685bf6d452bf2e7ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:55.500645 29578 raft_consensus.cc:1275] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:55.501999 29893 consensus_queue.cc:1048] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:02:55.513198 29726 raft_consensus.cc:1275] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:55.515663 29893 consensus_queue.cc:1048] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:02:55.524986 29578 raft_consensus.cc:1275] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:55.526242 29893 consensus_queue.cc:1048] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:02:55.536124 29726 raft_consensus.cc:1275] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:55.538360 29893 consensus_queue.cc:1048] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:02:55.588774 29726 raft_consensus.cc:1275] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:55.590175 29893 consensus_queue.cc:1048] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:02:55.600248 29578 raft_consensus.cc:1275] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:55.602242 29913 consensus_queue.cc:1048] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:02:55.719296 29726 raft_consensus.cc:1275] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:55.720538 29896 consensus_queue.cc:1048] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:02:55.729317 29578 raft_consensus.cc:1275] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:55.731418 29918 consensus_queue.cc:1048] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:02:55.755731 29578 raft_consensus.cc:1275] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:55.757062 29918 consensus_queue.cc:1048] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:55.766541 29726 raft_consensus.cc:1275] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Refusing update from remote peer e76383577c9d4685bf6d452bf2e7ec32: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:02:55.768013 29918 consensus_queue.cc:1048] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:02:58.018496 29443 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20251025 20:02:58.019464 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:58.027050 29652 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
mode: GRACEFUL
new_leader_uuid: "23b530a2476d464eb636e2f3ea2e1442"
from {username='slave'} at 127.0.0.1:35700
I20251025 20:02:58.027845 29652 raft_consensus.cc:606] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Received request to transfer leadership to TS 23b530a2476d464eb636e2f3ea2e1442
I20251025 20:02:58.030511 29652 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
mode: GRACEFUL
new_leader_uuid: "23b530a2476d464eb636e2f3ea2e1442"
from {username='slave'} at 127.0.0.1:35700
I20251025 20:02:58.030961 29652 raft_consensus.cc:606] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Received request to transfer leadership to TS 23b530a2476d464eb636e2f3ea2e1442
I20251025 20:02:58.033231 29652 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
mode: GRACEFUL
new_leader_uuid: "23b530a2476d464eb636e2f3ea2e1442"
from {username='slave'} at 127.0.0.1:35700
I20251025 20:02:58.033674 29652 raft_consensus.cc:606] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Received request to transfer leadership to TS 23b530a2476d464eb636e2f3ea2e1442
I20251025 20:02:58.035763 29652 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
mode: GRACEFUL
new_leader_uuid: "10924e710604471a813b87ff5bbcfb8f"
from {username='slave'} at 127.0.0.1:35700
I20251025 20:02:58.036223 29652 raft_consensus.cc:606] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Received request to transfer leadership to TS 10924e710604471a813b87ff5bbcfb8f
I20251025 20:02:58.038491 29652 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38"
dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
mode: GRACEFUL
new_leader_uuid: "10924e710604471a813b87ff5bbcfb8f"
from {username='slave'} at 127.0.0.1:35700
I20251025 20:02:58.039069 29652 raft_consensus.cc:606] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Received request to transfer leadership to TS 10924e710604471a813b87ff5bbcfb8f
I20251025 20:02:58.040087 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251025 20:02:58.040617 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:02:58.062644 29924 raft_consensus.cc:993] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32: : Instructing follower 23b530a2476d464eb636e2f3ea2e1442 to start an election
I20251025 20:02:58.063251 29922 raft_consensus.cc:1081] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Signalling peer 23b530a2476d464eb636e2f3ea2e1442 to start an election
I20251025 20:02:58.064987 29578 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea"
dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
from {username='slave'} at 127.0.0.1:50006
I20251025 20:02:58.065490 29578 raft_consensus.cc:493] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:58.065785 29578 raft_consensus.cc:3060] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:58.071723 29578 raft_consensus.cc:515] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.073823 29578 leader_election.cc:290] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 3 election: Requested vote from peers e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:58.075048 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:58.075186 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1cf1641f71ef4ece8e154d1dba151cea" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:58.075961 29652 raft_consensus.cc:3055] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:02:58.076092 29726 raft_consensus.cc:3060] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:58.076246 29652 raft_consensus.cc:740] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:58.076997 29652 consensus_queue.cc:260] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.078127 29652 raft_consensus.cc:3060] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:58.081539 29726 raft_consensus.cc:2468] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 3.
I20251025 20:02:58.082916 29543 leader_election.cc:304] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:58.083179 29652 raft_consensus.cc:2468] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 3.
I20251025 20:02:58.084743 29925 raft_consensus.cc:2804] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:02:58.085601 29925 raft_consensus.cc:697] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 3 LEADER]: Becoming Leader. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:58.086463 29925 consensus_queue.cc:237] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.094480 29475 catalog_manager.cc:5649] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 reported cstate change: term changed from 2 to 3, leader changed from e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194) to 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193). New cstate: current_term: 3 leader_uuid: "23b530a2476d464eb636e2f3ea2e1442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:58.194698 29924 raft_consensus.cc:993] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32: : Instructing follower 23b530a2476d464eb636e2f3ea2e1442 to start an election
I20251025 20:02:58.195169 29922 raft_consensus.cc:1081] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Signalling peer 23b530a2476d464eb636e2f3ea2e1442 to start an election
I20251025 20:02:58.196655 29576 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99"
dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
from {username='slave'} at 127.0.0.1:50006
I20251025 20:02:58.197096 29576 raft_consensus.cc:493] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:58.197322 29576 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:02:58.202585 29576 raft_consensus.cc:515] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.204125 29576 leader_election.cc:290] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 4 election: Requested vote from peers e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:58.205189 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:58.205185 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585e4a99f17f4f0e9b012f6cc74bdc99" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:58.205709 29652 raft_consensus.cc:3055] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:02:58.205765 29726 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:02:58.206039 29652 raft_consensus.cc:740] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:58.206743 29652 consensus_queue.cc:260] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.207743 29652 raft_consensus.cc:3060] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:02:58.210767 29726 raft_consensus.cc:2468] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 4.
I20251025 20:02:58.211735 29543 leader_election.cc:304] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:58.212417 29925 raft_consensus.cc:2804] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:02:58.212538 29652 raft_consensus.cc:2468] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 4.
I20251025 20:02:58.212819 29925 raft_consensus.cc:697] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 LEADER]: Becoming Leader. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:58.213773 29925 consensus_queue.cc:237] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.220935 29475 catalog_manager.cc:5649] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 reported cstate change: term changed from 3 to 4, leader changed from e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194) to 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "23b530a2476d464eb636e2f3ea2e1442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:58.290923 29922 raft_consensus.cc:993] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32: : Instructing follower 10924e710604471a813b87ff5bbcfb8f to start an election
I20251025 20:02:58.291446 29924 raft_consensus.cc:1081] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Signalling peer 10924e710604471a813b87ff5bbcfb8f to start an election
I20251025 20:02:58.292773 29726 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38"
dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
from {username='slave'} at 127.0.0.1:60918
I20251025 20:02:58.293176 29726 raft_consensus.cc:493] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:58.293408 29726 raft_consensus.cc:3060] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:58.297958 29726 raft_consensus.cc:515] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.299880 29726 leader_election.cc:290] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 3 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:58.301092 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:58.301776 29652 raft_consensus.cc:3055] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:02:58.301671 29576 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb7a820773d0484caab2a0e7a7bb9e38" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:58.302474 29652 raft_consensus.cc:740] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:58.302713 29576 raft_consensus.cc:3060] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:58.303253 29652 consensus_queue.cc:260] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.304347 29922 raft_consensus.cc:993] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32: : Instructing follower 23b530a2476d464eb636e2f3ea2e1442 to start an election
I20251025 20:02:58.304587 29652 raft_consensus.cc:3060] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:02:58.304832 29924 raft_consensus.cc:1081] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Signalling peer 23b530a2476d464eb636e2f3ea2e1442 to start an election
I20251025 20:02:58.306574 29578 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0"
dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
from {username='slave'} at 127.0.0.1:50006
I20251025 20:02:58.307122 29578 raft_consensus.cc:493] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:58.307439 29578 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:02:58.309073 29576 raft_consensus.cc:2468] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 3.
I20251025 20:02:58.310392 29691 leader_election.cc:304] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:58.311206 29891 raft_consensus.cc:2804] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:02:58.311553 29652 raft_consensus.cc:2468] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 3.
I20251025 20:02:58.313596 29891 raft_consensus.cc:697] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 3 LEADER]: Becoming Leader. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:58.314235 29578 raft_consensus.cc:515] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.315168 29891 consensus_queue.cc:237] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.316653 29578 leader_election.cc:290] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 4 election: Requested vote from peers e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239), 10924e710604471a813b87ff5bbcfb8f (127.28.192.195:45807)
I20251025 20:02:58.316921 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:58.317487 29652 raft_consensus.cc:3055] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:02:58.317603 29726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9893e4b3767a41909e80e29b83fce0b0" candidate_uuid: "23b530a2476d464eb636e2f3ea2e1442" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
I20251025 20:02:58.318217 29726 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:02:58.317766 29652 raft_consensus.cc:740] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:58.320716 29652 consensus_queue.cc:260] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.322062 29652 raft_consensus.cc:3060] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:02:58.324657 29475 catalog_manager.cc:5649] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f reported cstate change: term changed from 2 to 3, leader changed from e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194) to 10924e710604471a813b87ff5bbcfb8f (127.28.192.195). New cstate: current_term: 3 leader_uuid: "10924e710604471a813b87ff5bbcfb8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20251025 20:02:58.327008 29726 raft_consensus.cc:2468] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 4.
I20251025 20:02:58.328750 29543 leader_election.cc:304] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:58.329546 29925 raft_consensus.cc:2804] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:02:58.330077 29925 raft_consensus.cc:697] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 LEADER]: Becoming Leader. State: Replica: 23b530a2476d464eb636e2f3ea2e1442, State: Running, Role: LEADER
I20251025 20:02:58.330935 29925 consensus_queue.cc:237] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.332401 29652 raft_consensus.cc:2468] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b530a2476d464eb636e2f3ea2e1442 in term 4.
I20251025 20:02:58.339936 29475 catalog_manager.cc:5649] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 reported cstate change: term changed from 3 to 4, leader changed from e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194) to 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "23b530a2476d464eb636e2f3ea2e1442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20251025 20:02:58.521872 29922 raft_consensus.cc:993] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32: : Instructing follower 10924e710604471a813b87ff5bbcfb8f to start an election
I20251025 20:02:58.522383 29924 raft_consensus.cc:1081] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Signalling peer 10924e710604471a813b87ff5bbcfb8f to start an election
I20251025 20:02:58.523883 29726 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284"
dest_uuid: "10924e710604471a813b87ff5bbcfb8f"
from {username='slave'} at 127.0.0.1:60918
I20251025 20:02:58.524389 29726 raft_consensus.cc:493] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:02:58.524668 29726 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:02:58.529349 29726 raft_consensus.cc:515] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.531014 29726 leader_election.cc:290] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 4 election: Requested vote from peers 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283), e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194:43239)
I20251025 20:02:58.531948 29578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "23b530a2476d464eb636e2f3ea2e1442"
I20251025 20:02:58.532156 29652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c43e53a8c2594107bb43f93727e95284" candidate_uuid: "10924e710604471a813b87ff5bbcfb8f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e76383577c9d4685bf6d452bf2e7ec32"
I20251025 20:02:58.532471 29578 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:02:58.532642 29652 raft_consensus.cc:3055] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:02:58.532931 29652 raft_consensus.cc:740] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e76383577c9d4685bf6d452bf2e7ec32, State: Running, Role: LEADER
I20251025 20:02:58.533569 29652 consensus_queue.cc:260] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.534677 29652 raft_consensus.cc:3060] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:02:58.537809 29578 raft_consensus.cc:2468] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 4.
I20251025 20:02:58.539021 29691 leader_election.cc:304] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10924e710604471a813b87ff5bbcfb8f, 23b530a2476d464eb636e2f3ea2e1442; no voters:
I20251025 20:02:58.539752 29652 raft_consensus.cc:2468] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10924e710604471a813b87ff5bbcfb8f in term 4.
I20251025 20:02:58.539853 29891 raft_consensus.cc:2804] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:02:58.540354 29891 raft_consensus.cc:697] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 4 LEADER]: Becoming Leader. State: Replica: 10924e710604471a813b87ff5bbcfb8f, State: Running, Role: LEADER
I20251025 20:02:58.541182 29891 consensus_queue.cc:237] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } }
I20251025 20:02:58.548698 29475 catalog_manager.cc:5649] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f reported cstate change: term changed from 3 to 4, leader changed from e76383577c9d4685bf6d452bf2e7ec32 (127.28.192.194) to 10924e710604471a813b87ff5bbcfb8f (127.28.192.195). New cstate: current_term: 4 leader_uuid: "10924e710604471a813b87ff5bbcfb8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20251025 20:02:58.597412 29726 raft_consensus.cc:1275] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Refusing update from remote peer 23b530a2476d464eb636e2f3ea2e1442: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:58.598779 29925 consensus_queue.cc:1048] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:02:58.610496 29652 raft_consensus.cc:1275] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Refusing update from remote peer 23b530a2476d464eb636e2f3ea2e1442: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:58.614113 29925 consensus_queue.cc:1048] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:02:58.627339 29652 raft_consensus.cc:1275] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Refusing update from remote peer 23b530a2476d464eb636e2f3ea2e1442: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:02:58.627493 29726 raft_consensus.cc:1275] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Refusing update from remote peer 23b530a2476d464eb636e2f3ea2e1442: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:02:58.628777 29926 consensus_queue.cc:1048] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251025 20:02:58.630342 29939 consensus_queue.cc:1048] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251025 20:02:58.750850 29652 raft_consensus.cc:1275] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Refusing update from remote peer 10924e710604471a813b87ff5bbcfb8f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:58.752060 29891 consensus_queue.cc:1048] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:02:58.761840 29578 raft_consensus.cc:1275] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Refusing update from remote peer 10924e710604471a813b87ff5bbcfb8f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:02:58.763453 29891 consensus_queue.cc:1048] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:02:58.799650 29727 raft_consensus.cc:1275] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Refusing update from remote peer 23b530a2476d464eb636e2f3ea2e1442: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:02:58.801061 29925 consensus_queue.cc:1048] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10924e710604471a813b87ff5bbcfb8f" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:02:58.811250 29652 raft_consensus.cc:1275] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Refusing update from remote peer 23b530a2476d464eb636e2f3ea2e1442: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:02:58.812952 29925 consensus_queue.cc:1048] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251025 20:02:59.027593 29652 raft_consensus.cc:1275] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Refusing update from remote peer 10924e710604471a813b87ff5bbcfb8f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:02:59.028828 29891 consensus_queue.cc:1048] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76383577c9d4685bf6d452bf2e7ec32" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:02:59.040356 29578 raft_consensus.cc:1275] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 FOLLOWER]: Refusing update from remote peer 10924e710604471a813b87ff5bbcfb8f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:02:59.041992 29946 consensus_queue.cc:1048] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b530a2476d464eb636e2f3ea2e1442" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:02:59.042237 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:02:59.057343 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:02:59.057832 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:00.058938 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:00.064424 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:00.064900 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:01.065935 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:01.071414 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:01.071898 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:02.072878 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:02.078117 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:02.078590 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:03.079707 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:03.084826 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:03.085362 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:04.086562 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:04.092137 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:04.092615 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:05.093770 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:05.099359 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:05.099846 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:06.101202 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:06.107353 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:06.107816 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:07.375562 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:07.384228 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:07.384693 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251025 20:03:07.695921 29506 debug-util.cc:398] Leaking SignalData structure 0x7b0800061060 after lost signal to thread 29446
W20251025 20:03:07.696928 29506 debug-util.cc:398] Leaking SignalData structure 0x7b0800061740 after lost signal to thread 29509
W20251025 20:03:07.697996 29506 debug-util.cc:398] Leaking SignalData structure 0x7b0800156280 after lost signal to thread 29602
W20251025 20:03:07.699091 29506 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce4a0 after lost signal to thread 29677
W20251025 20:03:07.700050 29506 debug-util.cc:398] Leaking SignalData structure 0x7b08001d07a0 after lost signal to thread 29751
I20251025 20:03:08.418290 29443 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20251025 20:03:08.419015 29443 tablet_server.cc:178] TabletServer@127.28.192.193:0 shutting down...
W20251025 20:03:08.462805 29617 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251025 20:03:08.465267 29443 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251025 20:03:08.466619 29443 tablet_replica.cc:333] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442: stopping tablet replica
I20251025 20:03:08.467597 29443 raft_consensus.cc:2243] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.468549 29443 raft_consensus.cc:2272] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Raft consensus is shut down!
W20251025 20:03:08.469959 29617 consensus_peers.cc:597] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 -> Peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Couldn't send request to peer 23b530a2476d464eb636e2f3ea2e1442. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:08.472167 29443 tablet_replica.cc:333] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442: stopping tablet replica
I20251025 20:03:08.472807 29443 raft_consensus.cc:2243] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 3 LEADER]: Raft consensus shutting down.
I20251025 20:03:08.473752 29443 raft_consensus.cc:2272] T 1cf1641f71ef4ece8e154d1dba151cea P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.475962 29443 tablet_replica.cc:333] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442: stopping tablet replica
I20251025 20:03:08.476559 29443 raft_consensus.cc:2243] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.477118 29443 raft_consensus.cc:2272] T b067857e85d640b584839f045b12b019 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.479218 29443 tablet_replica.cc:333] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442: stopping tablet replica
I20251025 20:03:08.479897 29443 raft_consensus.cc:2243] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 LEADER]: Raft consensus shutting down.
I20251025 20:03:08.481068 29443 raft_consensus.cc:2272] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.483996 29443 tablet_replica.cc:333] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442: stopping tablet replica
I20251025 20:03:08.484553 29443 raft_consensus.cc:2243] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.485002 29443 raft_consensus.cc:2272] T 4a12402079514257b1bd8b657887b2bc P 23b530a2476d464eb636e2f3ea2e1442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.487088 29443 tablet_replica.cc:333] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442: stopping tablet replica
I20251025 20:03:08.487648 29443 raft_consensus.cc:2243] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 LEADER]: Raft consensus shutting down.
I20251025 20:03:08.488664 29443 raft_consensus.cc:2272] T 9893e4b3767a41909e80e29b83fce0b0 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.490682 29443 tablet_replica.cc:333] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442: stopping tablet replica
I20251025 20:03:08.491259 29443 raft_consensus.cc:2243] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.491834 29443 raft_consensus.cc:2272] T fb7a820773d0484caab2a0e7a7bb9e38 P 23b530a2476d464eb636e2f3ea2e1442 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.493866 29443 tablet_replica.cc:333] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442: stopping tablet replica
I20251025 20:03:08.494391 29443 raft_consensus.cc:2243] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.494918 29443 raft_consensus.cc:2272] T c43e53a8c2594107bb43f93727e95284 P 23b530a2476d464eb636e2f3ea2e1442 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.496942 29443 tablet_replica.cc:333] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442: stopping tablet replica
I20251025 20:03:08.497478 29443 raft_consensus.cc:2243] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.498013 29443 raft_consensus.cc:2272] T 7c443311caa74ff4a1df53c5323f21b2 P 23b530a2476d464eb636e2f3ea2e1442 [term 2 FOLLOWER]: Raft consensus is shut down!
W20251025 20:03:08.637274 29617 consensus_peers.cc:597] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 -> Peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Couldn't send request to peer 23b530a2476d464eb636e2f3ea2e1442. Status: Network error: Client connection negotiation failed: client connection to 127.28.192.193:36283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:08.637977 29443 tablet_server.cc:195] TabletServer@127.28.192.193:0 shutdown complete.
I20251025 20:03:08.673063 29443 tablet_server.cc:178] TabletServer@127.28.192.194:0 shutting down...
W20251025 20:03:08.686205 29691 consensus_peers.cc:597] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f -> Peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Couldn't send request to peer 23b530a2476d464eb636e2f3ea2e1442. Status: Network error: Client connection negotiation failed: client connection to 127.28.192.193:36283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:08.722007 29443 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251025 20:03:08.722884 29443 tablet_replica.cc:333] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32: stopping tablet replica
I20251025 20:03:08.723804 29443 raft_consensus.cc:2243] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.724391 29443 raft_consensus.cc:2272] T b067857e85d640b584839f045b12b019 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.726660 29443 tablet_replica.cc:333] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32: stopping tablet replica
I20251025 20:03:08.727257 29443 raft_consensus.cc:2243] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.727758 29443 raft_consensus.cc:2272] T c43e53a8c2594107bb43f93727e95284 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.729763 29443 tablet_replica.cc:333] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32: stopping tablet replica
I20251025 20:03:08.730417 29443 raft_consensus.cc:2243] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.730914 29443 raft_consensus.cc:2272] T 9893e4b3767a41909e80e29b83fce0b0 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.732968 29443 tablet_replica.cc:333] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32: stopping tablet replica
I20251025 20:03:08.733500 29443 raft_consensus.cc:2243] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.734004 29443 raft_consensus.cc:2272] T fb7a820773d0484caab2a0e7a7bb9e38 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.736133 29443 tablet_replica.cc:333] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32: stopping tablet replica
I20251025 20:03:08.736690 29443 raft_consensus.cc:2243] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.737151 29443 raft_consensus.cc:2272] T 1cf1641f71ef4ece8e154d1dba151cea P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.739158 29443 tablet_replica.cc:333] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32: stopping tablet replica
I20251025 20:03:08.739725 29443 raft_consensus.cc:2243] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 LEADER]: Raft consensus shutting down.
I20251025 20:03:08.740666 29443 raft_consensus.cc:2272] T 9ae63ee9cca14dae8be38ff2b05c01a6 P e76383577c9d4685bf6d452bf2e7ec32 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.742745 29443 tablet_replica.cc:333] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32: stopping tablet replica
I20251025 20:03:08.743258 29443 raft_consensus.cc:2243] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 LEADER]: Raft consensus shutting down.
I20251025 20:03:08.744150 29443 raft_consensus.cc:2272] T 7c443311caa74ff4a1df53c5323f21b2 P e76383577c9d4685bf6d452bf2e7ec32 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.746322 29443 tablet_replica.cc:333] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32: stopping tablet replica
I20251025 20:03:08.746874 29443 raft_consensus.cc:2243] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:03:08.747771 29443 raft_consensus.cc:2272] T 4a12402079514257b1bd8b657887b2bc P e76383577c9d4685bf6d452bf2e7ec32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.749818 29443 tablet_replica.cc:333] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32: stopping tablet replica
I20251025 20:03:08.750320 29443 raft_consensus.cc:2243] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.750815 29443 raft_consensus.cc:2272] T 585e4a99f17f4f0e9b012f6cc74bdc99 P e76383577c9d4685bf6d452bf2e7ec32 [term 4 FOLLOWER]: Raft consensus is shut down!
W20251025 20:03:08.782307 29691 consensus_peers.cc:597] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f -> Peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Couldn't send request to peer 23b530a2476d464eb636e2f3ea2e1442. Status: Network error: Client connection negotiation failed: client connection to 127.28.192.193:36283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:08.784117 29443 tablet_server.cc:195] TabletServer@127.28.192.194:0 shutdown complete.
I20251025 20:03:08.813711 29443 tablet_server.cc:178] TabletServer@127.28.192.195:0 shutting down...
W20251025 20:03:08.850529 29691 consensus_peers.cc:597] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f -> Peer 23b530a2476d464eb636e2f3ea2e1442 (127.28.192.193:36283): Couldn't send request to peer 23b530a2476d464eb636e2f3ea2e1442. Status: Network error: Client connection negotiation failed: client connection to 127.28.192.193:36283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:08.854446 29443 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251025 20:03:08.855381 29443 tablet_replica.cc:333] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f: stopping tablet replica
I20251025 20:03:08.856158 29443 raft_consensus.cc:2243] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.856689 29443 raft_consensus.cc:2272] T 4a12402079514257b1bd8b657887b2bc P 10924e710604471a813b87ff5bbcfb8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.858997 29443 tablet_replica.cc:333] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f: stopping tablet replica
I20251025 20:03:08.859571 29443 raft_consensus.cc:2243] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.860184 29443 raft_consensus.cc:2272] T 585e4a99f17f4f0e9b012f6cc74bdc99 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.862311 29443 tablet_replica.cc:333] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f: stopping tablet replica
I20251025 20:03:08.863106 29443 raft_consensus.cc:2243] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.863961 29443 raft_consensus.cc:2272] T 9893e4b3767a41909e80e29b83fce0b0 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.866134 29443 tablet_replica.cc:333] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f: stopping tablet replica
I20251025 20:03:08.866725 29443 raft_consensus.cc:2243] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 3 LEADER]: Raft consensus shutting down.
I20251025 20:03:08.867722 29443 raft_consensus.cc:2272] T fb7a820773d0484caab2a0e7a7bb9e38 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.869864 29443 tablet_replica.cc:333] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f: stopping tablet replica
I20251025 20:03:08.870479 29443 raft_consensus.cc:2243] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 2 LEADER]: Raft consensus shutting down.
I20251025 20:03:08.871404 29443 raft_consensus.cc:2272] T b067857e85d640b584839f045b12b019 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.873412 29443 tablet_replica.cc:333] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f: stopping tablet replica
I20251025 20:03:08.874019 29443 raft_consensus.cc:2243] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.874545 29443 raft_consensus.cc:2272] T 1cf1641f71ef4ece8e154d1dba151cea P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.876526 29443 tablet_replica.cc:333] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f: stopping tablet replica
I20251025 20:03:08.877071 29443 raft_consensus.cc:2243] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.877575 29443 raft_consensus.cc:2272] T 9ae63ee9cca14dae8be38ff2b05c01a6 P 10924e710604471a813b87ff5bbcfb8f [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.879693 29443 tablet_replica.cc:333] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f: stopping tablet replica
I20251025 20:03:08.880254 29443 raft_consensus.cc:2243] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:08.880700 29443 raft_consensus.cc:2272] T 7c443311caa74ff4a1df53c5323f21b2 P 10924e710604471a813b87ff5bbcfb8f [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.882642 29443 tablet_replica.cc:333] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f: stopping tablet replica
I20251025 20:03:08.883176 29443 raft_consensus.cc:2243] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 4 LEADER]: Raft consensus shutting down.
I20251025 20:03:08.884189 29443 raft_consensus.cc:2272] T c43e53a8c2594107bb43f93727e95284 P 10924e710604471a813b87ff5bbcfb8f [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:08.917400 29443 tablet_server.cc:195] TabletServer@127.28.192.195:0 shutdown complete.
I20251025 20:03:08.950201 29443 master.cc:561] Master@127.28.192.254:33913 shutting down...
I20251025 20:03:09.226965 29530 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20251025 20:03:09.249606 29443 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:03:09.250605 29443 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:09.251116 29443 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7857dfba33b246c99d5979bb2a34a369: stopping tablet replica
I20251025 20:03:10.261322 29443 master.cc:583] Master@127.28.192.254:33913 shutdown complete.
10924e710604471a813b87ff5bbcfb8f 1
23b530a2476d464eb636e2f3ea2e1442 4
e76383577c9d4685bf6d452bf2e7ec32 4
10924e710604471a813b87ff5bbcfb8f 3
23b530a2476d464eb636e2f3ea2e1442 3
e76383577c9d4685bf6d452bf2e7ec32 3
10924e710604471a813b87ff5bbcfb8f 1
23b530a2476d464eb636e2f3ea2e1442 0
e76383577c9d4685bf6d452bf2e7ec32 8
10924e710604471a813b87ff5bbcfb8f 3
23b530a2476d464eb636e2f3ea2e1442 3
e76383577c9d4685bf6d452bf2e7ec32 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (33240 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20251025 20:03:10.308241 29443 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.192.254:44611
I20251025 20:03:10.309597 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:03:10.315532 29970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:10.316519 29971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:10.318220 29973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:03:10.319413 29443 server_base.cc:1047] running on GCE node
I20251025 20:03:10.320582 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:03:10.320799 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:03:10.320966 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422590320944 us; error 0 us; skew 500 ppm
I20251025 20:03:10.321529 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:03:10.324353 29443 webserver.cc:492] Webserver started at http://127.28.192.254:39325/ using document root <none> and password file <none>
I20251025 20:03:10.324915 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:03:10.325101 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:03:10.325419 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:03:10.326848 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/master-0-root/instance:
uuid: "ef8be305b62d409cb81816f1f11d62f8"
format_stamp: "Formatted at 2025-10-25 20:03:10 on dist-test-slave-svnn"
I20251025 20:03:10.332219 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251025 20:03:10.336519 29978 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:10.337379 29443 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251025 20:03:10.337738 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/master-0-root
uuid: "ef8be305b62d409cb81816f1f11d62f8"
format_stamp: "Formatted at 2025-10-25 20:03:10 on dist-test-slave-svnn"
I20251025 20:03:10.338130 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:03:10.350653 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:03:10.352150 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:03:10.402441 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.254:44611
I20251025 20:03:10.402558 30029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.254:44611 every 8 connection(s)
I20251025 20:03:10.407018 30030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:10.418578 30030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8: Bootstrap starting.
I20251025 20:03:10.424639 30030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:10.429848 30030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8: No bootstrap required, opened a new log
I20251025 20:03:10.432507 30030 raft_consensus.cc:359] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8be305b62d409cb81816f1f11d62f8" member_type: VOTER }
I20251025 20:03:10.433010 30030 raft_consensus.cc:385] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:10.433305 30030 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8be305b62d409cb81816f1f11d62f8, State: Initialized, Role: FOLLOWER
I20251025 20:03:10.433995 30030 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8be305b62d409cb81816f1f11d62f8" member_type: VOTER }
I20251025 20:03:10.434520 30030 raft_consensus.cc:399] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:03:10.434769 30030 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:03:10.435045 30030 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:10.439781 30030 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8be305b62d409cb81816f1f11d62f8" member_type: VOTER }
I20251025 20:03:10.440475 30030 leader_election.cc:304] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef8be305b62d409cb81816f1f11d62f8; no voters:
I20251025 20:03:10.441830 30030 leader_election.cc:290] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:03:10.442368 30033 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:10.443909 30033 raft_consensus.cc:697] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 1 LEADER]: Becoming Leader. State: Replica: ef8be305b62d409cb81816f1f11d62f8, State: Running, Role: LEADER
I20251025 20:03:10.444738 30033 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8be305b62d409cb81816f1f11d62f8" member_type: VOTER }
I20251025 20:03:10.445254 30030 sys_catalog.cc:565] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:03:10.451009 30034 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef8be305b62d409cb81816f1f11d62f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8be305b62d409cb81816f1f11d62f8" member_type: VOTER } }
I20251025 20:03:10.451372 30035 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef8be305b62d409cb81816f1f11d62f8. Latest consensus state: current_term: 1 leader_uuid: "ef8be305b62d409cb81816f1f11d62f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8be305b62d409cb81816f1f11d62f8" member_type: VOTER } }
I20251025 20:03:10.452230 30034 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [sys.catalog]: This master's current role is: LEADER
I20251025 20:03:10.452256 30035 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [sys.catalog]: This master's current role is: LEADER
I20251025 20:03:10.457350 30041 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:03:10.466357 30041 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:03:10.467494 29443 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251025 20:03:10.476543 30041 catalog_manager.cc:1357] Generated new cluster ID: f1f9300eb34d4cb9bd4d563e713b5bfb
I20251025 20:03:10.476917 30041 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:03:10.503119 30041 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:03:10.504601 30041 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:03:10.527346 30041 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8: Generated new TSK 0
I20251025 20:03:10.528090 30041 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 20:03:10.534401 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:03:10.541143 30051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:10.542533 30052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:10.544078 30054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:03:10.544615 29443 server_base.cc:1047] running on GCE node
I20251025 20:03:10.545595 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:03:10.545796 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:03:10.545991 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422590545974 us; error 0 us; skew 500 ppm
I20251025 20:03:10.546516 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:03:10.549149 29443 webserver.cc:492] Webserver started at http://127.28.192.193:35713/ using document root <none> and password file <none>
I20251025 20:03:10.549765 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:03:10.550005 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:03:10.550285 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:03:10.551442 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-0-root/instance:
uuid: "be90690510254d95a22a7def29abc3c4"
format_stamp: "Formatted at 2025-10-25 20:03:10 on dist-test-slave-svnn"
I20251025 20:03:10.556541 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20251025 20:03:10.560606 30059 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:10.561676 29443 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251025 20:03:10.562080 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-0-root
uuid: "be90690510254d95a22a7def29abc3c4"
format_stamp: "Formatted at 2025-10-25 20:03:10 on dist-test-slave-svnn"
I20251025 20:03:10.562414 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:03:10.579362 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:03:10.580834 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:03:10.582561 29443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:03:10.585266 29443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:03:10.585505 29443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:10.585772 29443 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:03:10.585975 29443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20251025 20:03:10.632001 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.193:34415
I20251025 20:03:10.632086 30121 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.193:34415 every 8 connection(s)
I20251025 20:03:10.637197 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:03:10.646003 30126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:10.648317 30127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:10.654224 30129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:03:10.655153 29443 server_base.cc:1047] running on GCE node
I20251025 20:03:10.657018 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:03:10.657239 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:03:10.657445 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422590657423 us; error 0 us; skew 500 ppm
I20251025 20:03:10.658174 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:03:10.658267 30122 heartbeater.cc:344] Connected to a master server at 127.28.192.254:44611
I20251025 20:03:10.658737 30122 heartbeater.cc:461] Registering TS with master...
I20251025 20:03:10.659579 30122 heartbeater.cc:507] Master 127.28.192.254:44611 requested a full tablet report, sending...
I20251025 20:03:10.661514 29443 webserver.cc:492] Webserver started at http://127.28.192.194:35767/ using document root <none> and password file <none>
I20251025 20:03:10.662370 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:03:10.662254 29995 ts_manager.cc:194] Registered new tserver with Master: be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415)
I20251025 20:03:10.662709 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:03:10.663116 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:03:10.664950 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-1-root/instance:
uuid: "17d90a5990d3424db25cf46b32327df5"
format_stamp: "Formatted at 2025-10-25 20:03:10 on dist-test-slave-svnn"
I20251025 20:03:10.665169 29995 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36340
I20251025 20:03:10.670727 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20251025 20:03:10.674851 30134 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:10.675699 29443 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:03:10.676014 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-1-root
uuid: "17d90a5990d3424db25cf46b32327df5"
format_stamp: "Formatted at 2025-10-25 20:03:10 on dist-test-slave-svnn"
I20251025 20:03:10.676322 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:03:10.711257 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:03:10.712812 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:03:10.714735 29443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:03:10.717482 29443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:03:10.717689 29443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:10.717885 29443 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:03:10.718083 29443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251025 20:03:10.764791 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.194:44617
I20251025 20:03:10.764876 30196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.194:44617 every 8 connection(s)
I20251025 20:03:10.770205 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:03:10.777760 30200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:10.780402 30201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:10.784188 30203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:03:10.784747 29443 server_base.cc:1047] running on GCE node
I20251025 20:03:10.785254 30197 heartbeater.cc:344] Connected to a master server at 127.28.192.254:44611
I20251025 20:03:10.785741 30197 heartbeater.cc:461] Registering TS with master...
I20251025 20:03:10.785995 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:03:10.786281 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:03:10.786502 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422590786479 us; error 0 us; skew 500 ppm
I20251025 20:03:10.786875 30197 heartbeater.cc:507] Master 127.28.192.254:44611 requested a full tablet report, sending...
I20251025 20:03:10.787312 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:03:10.789633 29995 ts_manager.cc:194] Registered new tserver with Master: 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617)
I20251025 20:03:10.790459 29443 webserver.cc:492] Webserver started at http://127.28.192.195:34251/ using document root <none> and password file <none>
I20251025 20:03:10.791038 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:03:10.791242 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:03:10.791368 29995 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36344
I20251025 20:03:10.791532 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:03:10.793339 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-2-root/instance:
uuid: "d00e76da60c94a3482243bba69caeee7"
format_stamp: "Formatted at 2025-10-25 20:03:10 on dist-test-slave-svnn"
I20251025 20:03:10.798720 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20251025 20:03:10.802680 30208 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:10.803586 29443 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20251025 20:03:10.803917 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-2-root
uuid: "d00e76da60c94a3482243bba69caeee7"
format_stamp: "Formatted at 2025-10-25 20:03:10 on dist-test-slave-svnn"
I20251025 20:03:10.804242 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761422556872874-29443-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:03:10.824069 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:03:10.825511 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:03:10.827248 29443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:03:10.829825 29443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:03:10.830099 29443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251025 20:03:10.830363 29443 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:03:10.830534 29443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:10.877336 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.195:36453
I20251025 20:03:10.877439 30270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.195:36453 every 8 connection(s)
I20251025 20:03:10.892585 30271 heartbeater.cc:344] Connected to a master server at 127.28.192.254:44611
I20251025 20:03:10.893002 30271 heartbeater.cc:461] Registering TS with master...
I20251025 20:03:10.893865 30271 heartbeater.cc:507] Master 127.28.192.254:44611 requested a full tablet report, sending...
I20251025 20:03:10.896142 29995 ts_manager.cc:194] Registered new tserver with Master: d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:10.896809 29443 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016019009s
I20251025 20:03:10.897594 29995 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36356
I20251025 20:03:10.899780 29443 test_util.cc:276] Using random seed: 668235302
I20251025 20:03:10.926481 29995 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36364:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:03:10.929843 29995 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:03:11.012715 30159 tablet_service.cc:1505] Processing CreateTablet for tablet 2038c9436df6436d92f7a25bfc5474bf (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251025 20:03:11.014014 30161 tablet_service.cc:1505] Processing CreateTablet for tablet ea6745d0705144b0999e3a2f7a35e8da (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251025 20:03:11.014612 30159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2038c9436df6436d92f7a25bfc5474bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.015578 30161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6745d0705144b0999e3a2f7a35e8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.019374 30160 tablet_service.cc:1505] Processing CreateTablet for tablet 38e64303be304c388287465e021ab853 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251025 20:03:11.021065 30160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e64303be304c388287465e021ab853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.012715 30162 tablet_service.cc:1505] Processing CreateTablet for tablet 728e3db5ac604f77b56f8e45c9520bf7 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251025 20:03:11.025316 30162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728e3db5ac604f77b56f8e45c9520bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.028339 30236 tablet_service.cc:1505] Processing CreateTablet for tablet 728e3db5ac604f77b56f8e45c9520bf7 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251025 20:03:11.029204 30235 tablet_service.cc:1505] Processing CreateTablet for tablet ea6745d0705144b0999e3a2f7a35e8da (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251025 20:03:11.034549 30234 tablet_service.cc:1505] Processing CreateTablet for tablet 38e64303be304c388287465e021ab853 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251025 20:03:11.031422 30158 tablet_service.cc:1505] Processing CreateTablet for tablet ceb28906ae414e8dac5ceea5f98ae1a8 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251025 20:03:11.037469 30158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb28906ae414e8dac5ceea5f98ae1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.037750 30233 tablet_service.cc:1505] Processing CreateTablet for tablet 2038c9436df6436d92f7a25bfc5474bf (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251025 20:03:11.030109 30236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728e3db5ac604f77b56f8e45c9520bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.040077 30233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2038c9436df6436d92f7a25bfc5474bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.040324 30157 tablet_service.cc:1505] Processing CreateTablet for tablet df3103d1e0b44cd79fc56a331af83528 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251025 20:03:11.042071 30234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e64303be304c388287465e021ab853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.043169 30156 tablet_service.cc:1505] Processing CreateTablet for tablet 45e270333ba34fd78b5b7c96a303e33b (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251025 20:03:11.044610 30157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3103d1e0b44cd79fc56a331af83528. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.047338 30156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45e270333ba34fd78b5b7c96a303e33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.051276 30235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6745d0705144b0999e3a2f7a35e8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.064193 30232 tablet_service.cc:1505] Processing CreateTablet for tablet ceb28906ae414e8dac5ceea5f98ae1a8 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251025 20:03:11.064527 30231 tablet_service.cc:1505] Processing CreateTablet for tablet df3103d1e0b44cd79fc56a331af83528 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251025 20:03:11.065852 30232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb28906ae414e8dac5ceea5f98ae1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.068068 30231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3103d1e0b44cd79fc56a331af83528. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.073465 30155 tablet_service.cc:1505] Processing CreateTablet for tablet 5961f6062df94219941afc0503ff3033 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251025 20:03:11.075196 30155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5961f6062df94219941afc0503ff3033. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.076125 30230 tablet_service.cc:1505] Processing CreateTablet for tablet 45e270333ba34fd78b5b7c96a303e33b (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251025 20:03:11.077927 30230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45e270333ba34fd78b5b7c96a303e33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.079035 30229 tablet_service.cc:1505] Processing CreateTablet for tablet 5961f6062df94219941afc0503ff3033 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251025 20:03:11.080458 30229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5961f6062df94219941afc0503ff3033. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.084275 30285 tablet_bootstrap.cc:492] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5: Bootstrap starting.
I20251025 20:03:11.099613 30154 tablet_service.cc:1505] Processing CreateTablet for tablet 4128ceefc2c541ccb009b58235c071e6 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251025 20:03:11.100687 30228 tablet_service.cc:1505] Processing CreateTablet for tablet 4128ceefc2c541ccb009b58235c071e6 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251025 20:03:11.101308 30154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4128ceefc2c541ccb009b58235c071e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.102227 30228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4128ceefc2c541ccb009b58235c071e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.138911 30087 tablet_service.cc:1505] Processing CreateTablet for tablet 728e3db5ac604f77b56f8e45c9520bf7 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251025 20:03:11.140520 30087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728e3db5ac604f77b56f8e45c9520bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.143365 30084 tablet_service.cc:1505] Processing CreateTablet for tablet 2038c9436df6436d92f7a25bfc5474bf (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251025 20:03:11.144950 30084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2038c9436df6436d92f7a25bfc5474bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.147913 30086 tablet_service.cc:1505] Processing CreateTablet for tablet ea6745d0705144b0999e3a2f7a35e8da (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251025 20:03:11.149366 30086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6745d0705144b0999e3a2f7a35e8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.151491 30081 tablet_service.cc:1505] Processing CreateTablet for tablet 45e270333ba34fd78b5b7c96a303e33b (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251025 20:03:11.152236 30085 tablet_service.cc:1505] Processing CreateTablet for tablet 38e64303be304c388287465e021ab853 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251025 20:03:11.153033 30081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45e270333ba34fd78b5b7c96a303e33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.154623 30227 tablet_service.cc:1505] Processing CreateTablet for tablet 7ce5274ff7fe43fab967b217512f197d (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251025 20:03:11.156140 30227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ce5274ff7fe43fab967b217512f197d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.160203 30083 tablet_service.cc:1505] Processing CreateTablet for tablet ceb28906ae414e8dac5ceea5f98ae1a8 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251025 20:03:11.158380 30285 tablet_bootstrap.cc:654] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.160945 30085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e64303be304c388287465e021ab853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.162364 30159 tablet_service.cc:1505] Processing CreateTablet for tablet 7ce5274ff7fe43fab967b217512f197d (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251025 20:03:11.164120 30082 tablet_service.cc:1505] Processing CreateTablet for tablet df3103d1e0b44cd79fc56a331af83528 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251025 20:03:11.165589 30082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3103d1e0b44cd79fc56a331af83528. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.165323 30080 tablet_service.cc:1505] Processing CreateTablet for tablet 5961f6062df94219941afc0503ff3033 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251025 20:03:11.166815 30080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5961f6062df94219941afc0503ff3033. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.167480 30083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb28906ae414e8dac5ceea5f98ae1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.171833 30079 tablet_service.cc:1505] Processing CreateTablet for tablet 4128ceefc2c541ccb009b58235c071e6 (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251025 20:03:11.174129 30159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ce5274ff7fe43fab967b217512f197d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.176280 30079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4128ceefc2c541ccb009b58235c071e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.191768 30078 tablet_service.cc:1505] Processing CreateTablet for tablet 7ce5274ff7fe43fab967b217512f197d (DEFAULT_TABLE table=test-workload [id=94b426182b8d42feadac797ac0488abe]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251025 20:03:11.193071 30078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ce5274ff7fe43fab967b217512f197d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:11.220445 30286 tablet_bootstrap.cc:492] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7: Bootstrap starting.
I20251025 20:03:11.228128 30287 tablet_bootstrap.cc:492] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4: Bootstrap starting.
I20251025 20:03:11.246153 30287 tablet_bootstrap.cc:654] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.249624 30286 tablet_bootstrap.cc:654] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.272878 30285 tablet_bootstrap.cc:492] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5: No bootstrap required, opened a new log
I20251025 20:03:11.273501 30285 ts_tablet_manager.cc:1403] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5: Time spent bootstrapping tablet: real 0.190s user 0.031s sys 0.033s
I20251025 20:03:11.276861 30285 raft_consensus.cc:359] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.278206 30285 raft_consensus.cc:385] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.278631 30285 raft_consensus.cc:740] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.279699 30285 consensus_queue.cc:260] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.287459 30286 tablet_bootstrap.cc:492] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7: No bootstrap required, opened a new log
I20251025 20:03:11.288446 30286 ts_tablet_manager.cc:1403] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.007s
I20251025 20:03:11.292805 30286 raft_consensus.cc:359] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.294471 30286 raft_consensus.cc:385] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.295006 30286 raft_consensus.cc:740] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.296393 30286 consensus_queue.cc:260] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.311836 30271 heartbeater.cc:499] Master 127.28.192.254:44611 was elected leader, sending a full tablet report...
I20251025 20:03:11.313167 30286 ts_tablet_manager.cc:1434] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7: Time spent starting tablet: real 0.024s user 0.014s sys 0.003s
I20251025 20:03:11.314335 30286 tablet_bootstrap.cc:492] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7: Bootstrap starting.
I20251025 20:03:11.323035 30286 tablet_bootstrap.cc:654] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.336014 30286 tablet_bootstrap.cc:492] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7: No bootstrap required, opened a new log
I20251025 20:03:11.336704 30286 ts_tablet_manager.cc:1403] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.017s
I20251025 20:03:11.340344 30286 raft_consensus.cc:359] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.341195 30286 raft_consensus.cc:385] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.341516 30286 raft_consensus.cc:740] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.343107 30287 tablet_bootstrap.cc:492] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4: No bootstrap required, opened a new log
I20251025 20:03:11.342645 30286 consensus_queue.cc:260] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.343822 30287 ts_tablet_manager.cc:1403] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4: Time spent bootstrapping tablet: real 0.116s user 0.023s sys 0.044s
I20251025 20:03:11.345600 30286 ts_tablet_manager.cc:1434] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20251025 20:03:11.346974 30286 tablet_bootstrap.cc:492] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7: Bootstrap starting.
I20251025 20:03:11.348615 30287 raft_consensus.cc:359] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.351559 30287 raft_consensus.cc:385] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.354841 30197 heartbeater.cc:499] Master 127.28.192.254:44611 was elected leader, sending a full tablet report...
I20251025 20:03:11.355912 30285 ts_tablet_manager.cc:1434] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5: Time spent starting tablet: real 0.082s user 0.027s sys 0.045s
I20251025 20:03:11.356781 30286 tablet_bootstrap.cc:654] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.356829 30292 raft_consensus.cc:493] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.357419 30292 raft_consensus.cc:515] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.360258 30292 leader_election.cc:290] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617)
I20251025 20:03:11.361795 30285 tablet_bootstrap.cc:492] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5: Bootstrap starting.
I20251025 20:03:11.369524 30285 tablet_bootstrap.cc:654] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.351954 30287 raft_consensus.cc:740] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.377784 30287 consensus_queue.cc:260] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
W20251025 20:03:11.381880 30272 tablet.cc:2378] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:03:11.384555 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
W20251025 20:03:11.387337 30209 leader_election.cc:343] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.397748 30122 heartbeater.cc:499] Master 127.28.192.254:44611 was elected leader, sending a full tablet report...
I20251025 20:03:11.398717 30285 tablet_bootstrap.cc:492] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5: No bootstrap required, opened a new log
I20251025 20:03:11.398797 30287 ts_tablet_manager.cc:1434] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4: Time spent starting tablet: real 0.055s user 0.006s sys 0.007s
I20251025 20:03:11.399672 30285 ts_tablet_manager.cc:1403] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.020s
I20251025 20:03:11.400727 30287 tablet_bootstrap.cc:492] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4: Bootstrap starting.
I20251025 20:03:11.402493 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5" is_pre_election: true
I20251025 20:03:11.403308 30285 raft_consensus.cc:359] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:11.404137 30285 raft_consensus.cc:385] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.404474 30285 raft_consensus.cc:740] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Initialized, Role: FOLLOWER
W20251025 20:03:11.404454 30211 leader_election.cc:343] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.405198 30211 leader_election.cc:304] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d00e76da60c94a3482243bba69caeee7; no voters: 17d90a5990d3424db25cf46b32327df5, be90690510254d95a22a7def29abc3c4
I20251025 20:03:11.405495 30285 consensus_queue.cc:260] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:11.406613 30292 raft_consensus.cc:2749] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:11.408361 30287 tablet_bootstrap.cc:654] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.408846 30285 ts_tablet_manager.cc:1434] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20251025 20:03:11.408871 30292 raft_consensus.cc:493] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.409538 30292 raft_consensus.cc:515] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.410171 30285 tablet_bootstrap.cc:492] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5: Bootstrap starting.
I20251025 20:03:11.412739 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2038c9436df6436d92f7a25bfc5474bf" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
I20251025 20:03:11.414098 30292 leader_election.cc:290] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617)
W20251025 20:03:11.414281 30209 leader_election.cc:343] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.415964 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2038c9436df6436d92f7a25bfc5474bf" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5" is_pre_election: true
I20251025 20:03:11.419013 30285 tablet_bootstrap.cc:654] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:11.419449 30211 leader_election.cc:343] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.420011 30211 leader_election.cc:304] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d00e76da60c94a3482243bba69caeee7; no voters: 17d90a5990d3424db25cf46b32327df5, be90690510254d95a22a7def29abc3c4
I20251025 20:03:11.420715 30286 tablet_bootstrap.cc:492] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7: No bootstrap required, opened a new log
I20251025 20:03:11.421286 30286 ts_tablet_manager.cc:1403] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7: Time spent bootstrapping tablet: real 0.075s user 0.015s sys 0.001s
I20251025 20:03:11.421299 30292 raft_consensus.cc:2749] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:11.424757 30286 raft_consensus.cc:359] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.425856 30286 raft_consensus.cc:385] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.426259 30286 raft_consensus.cc:740] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.427175 30286 consensus_queue.cc:260] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.431921 30296 raft_consensus.cc:493] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.432615 30286 ts_tablet_manager.cc:1434] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7: Time spent starting tablet: real 0.011s user 0.003s sys 0.004s
I20251025 20:03:11.432546 30296 raft_consensus.cc:515] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.434055 30286 tablet_bootstrap.cc:492] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7: Bootstrap starting.
I20251025 20:03:11.437206 30296 leader_election.cc:290] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.453842 30287 tablet_bootstrap.cc:492] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4: No bootstrap required, opened a new log
I20251025 20:03:11.454519 30287 ts_tablet_manager.cc:1403] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4: Time spent bootstrapping tablet: real 0.054s user 0.016s sys 0.020s
I20251025 20:03:11.455904 30285 tablet_bootstrap.cc:492] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5: No bootstrap required, opened a new log
I20251025 20:03:11.456425 30286 tablet_bootstrap.cc:654] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.456491 30285 ts_tablet_manager.cc:1403] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5: Time spent bootstrapping tablet: real 0.047s user 0.021s sys 0.018s
I20251025 20:03:11.456650 30291 raft_consensus.cc:493] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.457279 30291 raft_consensus.cc:515] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
W20251025 20:03:11.459188 30043 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:11.460263 30285 raft_consensus.cc:359] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.461123 30285 raft_consensus.cc:385] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.461532 30285 raft_consensus.cc:740] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.464102 30285 consensus_queue.cc:260] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.463714 30291 leader_election.cc:290] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453), be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415)
I20251025 20:03:11.468446 30287 raft_consensus.cc:359] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.469679 30287 raft_consensus.cc:385] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.474311 30287 raft_consensus.cc:740] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.475530 30287 consensus_queue.cc:260] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.493055 30287 ts_tablet_manager.cc:1434] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4: Time spent starting tablet: real 0.038s user 0.007s sys 0.003s
I20251025 20:03:11.494293 30287 tablet_bootstrap.cc:492] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4: Bootstrap starting.
I20251025 20:03:11.500314 30286 tablet_bootstrap.cc:492] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7: No bootstrap required, opened a new log
I20251025 20:03:11.501008 30286 ts_tablet_manager.cc:1403] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.004s
I20251025 20:03:11.501214 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4128ceefc2c541ccb009b58235c071e6" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:11.504999 30286 raft_consensus.cc:359] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.505797 30285 ts_tablet_manager.cc:1434] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5: Time spent starting tablet: real 0.049s user 0.018s sys 0.018s
I20251025 20:03:11.506172 30287 tablet_bootstrap.cc:654] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.507370 30285 tablet_bootstrap.cc:492] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5: Bootstrap starting.
I20251025 20:03:11.514909 30285 tablet_bootstrap.cc:654] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.505983 30286 raft_consensus.cc:385] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.515527 30286 raft_consensus.cc:740] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.516490 30286 consensus_queue.cc:260] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.517751 30246 raft_consensus.cc:2468] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 0.
I20251025 20:03:11.521462 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4128ceefc2c541ccb009b58235c071e6" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5" is_pre_election: true
I20251025 20:03:11.522210 30286 ts_tablet_manager.cc:1434] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7: Time spent starting tablet: real 0.021s user 0.005s sys 0.006s
W20251025 20:03:11.523260 30062 leader_election.cc:343] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.524468 30286 tablet_bootstrap.cc:492] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7: Bootstrap starting.
I20251025 20:03:11.525538 30060 leader_election.cc:304] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters: 17d90a5990d3424db25cf46b32327df5
I20251025 20:03:11.527441 30296 raft_consensus.cc:2804] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:11.527982 30296 raft_consensus.cc:493] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:11.528432 30296 raft_consensus.cc:3060] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.534103 30286 tablet_bootstrap.cc:654] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.539026 30285 tablet_bootstrap.cc:492] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5: No bootstrap required, opened a new log
I20251025 20:03:11.539580 30285 ts_tablet_manager.cc:1403] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.008s
I20251025 20:03:11.542129 30286 tablet_bootstrap.cc:492] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7: No bootstrap required, opened a new log
I20251025 20:03:11.542980 30286 ts_tablet_manager.cc:1403] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.011s
I20251025 20:03:11.542869 30285 raft_consensus.cc:359] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.543746 30285 raft_consensus.cc:385] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.544106 30285 raft_consensus.cc:740] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.544997 30285 consensus_queue.cc:260] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.546919 30286 raft_consensus.cc:359] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.547760 30286 raft_consensus.cc:385] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.548130 30286 raft_consensus.cc:740] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.548962 30286 consensus_queue.cc:260] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.551657 30286 ts_tablet_manager.cc:1434] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251025 20:03:11.552979 30286 tablet_bootstrap.cc:492] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7: Bootstrap starting.
I20251025 20:03:11.556224 30285 ts_tablet_manager.cc:1434] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20251025 20:03:11.562107 30285 tablet_bootstrap.cc:492] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5: Bootstrap starting.
I20251025 20:03:11.561945 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
W20251025 20:03:11.563913 30135 leader_election.cc:343] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.560669 30286 tablet_bootstrap.cc:654] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.568588 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:11.575062 30285 tablet_bootstrap.cc:654] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:11.577392 30135 leader_election.cc:343] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.581051 30135 leader_election.cc:304] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5; no voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7
I20251025 20:03:11.582440 30304 raft_consensus.cc:2749] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:11.562544 30296 raft_consensus.cc:515] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.587291 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4128ceefc2c541ccb009b58235c071e6" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:11.589368 30296 leader_election.cc:290] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
W20251025 20:03:11.589505 30062 leader_election.cc:343] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.590585 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4128ceefc2c541ccb009b58235c071e6" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:11.591292 30246 raft_consensus.cc:3060] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.592192 30286 tablet_bootstrap.cc:492] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7: No bootstrap required, opened a new log
I20251025 20:03:11.592561 30287 tablet_bootstrap.cc:492] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4: No bootstrap required, opened a new log
I20251025 20:03:11.592741 30286 ts_tablet_manager.cc:1403] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7: Time spent bootstrapping tablet: real 0.040s user 0.016s sys 0.015s
I20251025 20:03:11.592990 30287 ts_tablet_manager.cc:1403] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4: Time spent bootstrapping tablet: real 0.099s user 0.033s sys 0.053s
I20251025 20:03:11.596271 30285 tablet_bootstrap.cc:492] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5: No bootstrap required, opened a new log
I20251025 20:03:11.596078 30286 raft_consensus.cc:359] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.596284 30287 raft_consensus.cc:359] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.596980 30286 raft_consensus.cc:385] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.597199 30287 raft_consensus.cc:385] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.597247 30285 ts_tablet_manager.cc:1403] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.002s
I20251025 20:03:11.597350 30286 raft_consensus.cc:740] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.597673 30287 raft_consensus.cc:740] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.598490 30286 consensus_queue.cc:260] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.598542 30287 consensus_queue.cc:260] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.601437 30286 ts_tablet_manager.cc:1434] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20251025 20:03:11.601275 30285 raft_consensus.cc:359] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.602380 30285 raft_consensus.cc:385] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.602608 30246 raft_consensus.cc:2468] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 1.
I20251025 20:03:11.602752 30285 raft_consensus.cc:740] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.602698 30286 tablet_bootstrap.cc:492] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7: Bootstrap starting.
I20251025 20:03:11.603807 30060 leader_election.cc:304] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters: 17d90a5990d3424db25cf46b32327df5
I20251025 20:03:11.606050 30296 raft_consensus.cc:2804] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:11.606637 30296 raft_consensus.cc:697] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:11.607615 30296 consensus_queue.cc:237] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.610489 30285 consensus_queue.cc:260] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.612520 30286 tablet_bootstrap.cc:654] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.611969 30287 ts_tablet_manager.cc:1434] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20251025 20:03:11.613698 30287 tablet_bootstrap.cc:492] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4: Bootstrap starting.
I20251025 20:03:11.616042 30285 ts_tablet_manager.cc:1434] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20251025 20:03:11.623373 30285 tablet_bootstrap.cc:492] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5: Bootstrap starting.
I20251025 20:03:11.623065 30292 raft_consensus.cc:493] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.625648 30292 raft_consensus.cc:515] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.631588 30287 tablet_bootstrap.cc:654] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.631758 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
I20251025 20:03:11.641777 30097 raft_consensus.cc:2468] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 0.
I20251025 20:03:11.624054 30304 raft_consensus.cc:493] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.643481 30304 raft_consensus.cc:515] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.646843 30209 leader_election.cc:304] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:11.632761 30292 leader_election.cc:290] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617)
I20251025 20:03:11.655048 30292 raft_consensus.cc:2804] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:11.655670 30292 raft_consensus.cc:493] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:11.656137 30292 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:11.665390 30060 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:36453, user_credentials={real_user=slave}} blocked reactor thread for 61660.3us
I20251025 20:03:11.673014 30291 raft_consensus.cc:493] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.673676 30291 raft_consensus.cc:515] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.676266 30291 leader_election.cc:290] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.676827 30304 leader_election.cc:290] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.677673 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5" is_pre_election: true
I20251025 20:03:11.679097 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38e64303be304c388287465e021ab853" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:11.680020 30246 raft_consensus.cc:2468] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 0.
W20251025 20:03:11.682269 30211 leader_election.cc:343] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.682384 30292 raft_consensus.cc:515] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.683175 30245 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2038c9436df6436d92f7a25bfc5474bf" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:11.683318 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2038c9436df6436d92f7a25bfc5474bf" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
I20251025 20:03:11.683907 30245 raft_consensus.cc:2468] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 0.
I20251025 20:03:11.681689 30135 leader_election.cc:304] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:11.684544 30292 leader_election.cc:290] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617)
I20251025 20:03:11.685425 30304 raft_consensus.cc:2804] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:11.685824 30135 leader_election.cc:304] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:11.687058 30291 raft_consensus.cc:2804] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:11.687486 30291 raft_consensus.cc:493] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:11.687943 30291 raft_consensus.cc:3060] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.688549 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:11.689189 30097 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.690462 30285 tablet_bootstrap.cc:654] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.686021 30304 raft_consensus.cc:493] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:11.687507 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:11.694195 30304 raft_consensus.cc:3060] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:11.695902 30211 leader_election.cc:343] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.687032 29993 catalog_manager.cc:5649] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
W20251025 20:03:11.702711 30135 leader_election.cc:343] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.704100 30291 raft_consensus.cc:515] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.705760 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38e64303be304c388287465e021ab853" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
I20251025 20:03:11.708245 30291 leader_election.cc:290] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.709342 30245 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2038c9436df6436d92f7a25bfc5474bf" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:11.710209 30245 raft_consensus.cc:3060] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:11.710489 30135 leader_election.cc:343] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.714766 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2038c9436df6436d92f7a25bfc5474bf" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:11.715723 30286 tablet_bootstrap.cc:492] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7: No bootstrap required, opened a new log
I20251025 20:03:11.716275 30286 ts_tablet_manager.cc:1403] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7: Time spent bootstrapping tablet: real 0.114s user 0.010s sys 0.028s
I20251025 20:03:11.716420 30097 raft_consensus.cc:2468] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 1.
I20251025 20:03:11.717868 30209 leader_election.cc:304] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters: 17d90a5990d3424db25cf46b32327df5
I20251025 20:03:11.719095 30292 raft_consensus.cc:2804] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:11.719640 30292 raft_consensus.cc:697] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 1 LEADER]: Becoming Leader. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:11.720584 30292 consensus_queue.cc:237] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.721813 30286 raft_consensus.cc:359] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.722584 30286 raft_consensus.cc:385] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.722872 30286 raft_consensus.cc:740] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.723582 30286 consensus_queue.cc:260] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
W20251025 20:03:11.724053 30135 leader_election.cc:343] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.726473 30286 ts_tablet_manager.cc:1434] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20251025 20:03:11.726878 30245 raft_consensus.cc:2468] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 1.
I20251025 20:03:11.727602 30286 tablet_bootstrap.cc:492] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7: Bootstrap starting.
I20251025 20:03:11.728755 30135 leader_election.cc:304] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters: be90690510254d95a22a7def29abc3c4
I20251025 20:03:11.729866 30291 raft_consensus.cc:2804] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:11.731361 30291 raft_consensus.cc:697] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Becoming Leader. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:11.732909 30291 consensus_queue.cc:237] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.734117 30287 tablet_bootstrap.cc:492] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4: No bootstrap required, opened a new log
I20251025 20:03:11.735127 30287 ts_tablet_manager.cc:1403] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4: Time spent bootstrapping tablet: real 0.122s user 0.025s sys 0.002s
I20251025 20:03:11.736490 30286 tablet_bootstrap.cc:654] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.736775 30285 tablet_bootstrap.cc:492] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5: No bootstrap required, opened a new log
I20251025 20:03:11.737370 30285 ts_tablet_manager.cc:1403] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5: Time spent bootstrapping tablet: real 0.114s user 0.018s sys 0.008s
I20251025 20:03:11.739593 30287 raft_consensus.cc:359] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:11.740854 30287 raft_consensus.cc:385] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.741287 30287 raft_consensus.cc:740] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.741369 30285 raft_consensus.cc:359] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.742448 30285 raft_consensus.cc:385] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.742561 30287 consensus_queue.cc:260] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:11.745078 30285 raft_consensus.cc:740] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.747850 30285 consensus_queue.cc:260] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.748276 30287 ts_tablet_manager.cc:1434] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20251025 20:03:11.749503 30286 tablet_bootstrap.cc:492] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7: No bootstrap required, opened a new log
I20251025 20:03:11.750084 30286 ts_tablet_manager.cc:1403] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.006s
I20251025 20:03:11.724490 30304 raft_consensus.cc:515] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.750372 30287 tablet_bootstrap.cc:492] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4: Bootstrap starting.
I20251025 20:03:11.753526 30286 raft_consensus.cc:359] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.754290 30286 raft_consensus.cc:385] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.754592 30286 raft_consensus.cc:740] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.755578 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38e64303be304c388287465e021ab853" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:11.755414 30286 consensus_queue.cc:260] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.757550 30285 ts_tablet_manager.cc:1434] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5: Time spent starting tablet: real 0.020s user 0.011s sys 0.000s
I20251025 20:03:11.757439 30245 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38e64303be304c388287465e021ab853" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:11.758277 30286 ts_tablet_manager.cc:1434] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251025 20:03:11.758188 30245 raft_consensus.cc:3060] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.758778 30285 tablet_bootstrap.cc:492] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5: Bootstrap starting.
W20251025 20:03:11.759584 30135 leader_election.cc:343] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.762014 30304 leader_election.cc:290] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.766319 30287 tablet_bootstrap.cc:654] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.767194 30245 raft_consensus.cc:2468] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 1.
I20251025 20:03:11.768560 30135 leader_election.cc:304] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters: be90690510254d95a22a7def29abc3c4
I20251025 20:03:11.768975 30286 tablet_bootstrap.cc:492] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7: Bootstrap starting.
I20251025 20:03:11.769502 30304 raft_consensus.cc:2804] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:11.770507 30304 raft_consensus.cc:697] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Becoming Leader. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:11.771561 30304 consensus_queue.cc:237] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.777612 30286 tablet_bootstrap.cc:654] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.782774 30287 tablet_bootstrap.cc:492] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4: No bootstrap required, opened a new log
I20251025 20:03:11.777107 29995 catalog_manager.cc:5649] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d90a5990d3424db25cf46b32327df5 (127.28.192.194). New cstate: current_term: 1 leader_uuid: "17d90a5990d3424db25cf46b32327df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:11.781646 29994 catalog_manager.cc:5649] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d00e76da60c94a3482243bba69caeee7 (127.28.192.195). New cstate: current_term: 1 leader_uuid: "d00e76da60c94a3482243bba69caeee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:11.785832 30312 raft_consensus.cc:493] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.786476 30312 raft_consensus.cc:515] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.790506 30245 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ce5274ff7fe43fab967b217512f197d" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:11.791471 30245 raft_consensus.cc:2468] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 0.
I20251025 20:03:11.792721 30312 leader_election.cc:290] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.794586 30060 leader_election.cc:304] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:11.795246 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ce5274ff7fe43fab967b217512f197d" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5" is_pre_election: true
I20251025 20:03:11.796303 30312 raft_consensus.cc:2804] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:11.796756 30312 raft_consensus.cc:493] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:11.797210 30312 raft_consensus.cc:3060] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:11.797256 30062 leader_election.cc:343] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.798903 30285 tablet_bootstrap.cc:654] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.801450 30287 ts_tablet_manager.cc:1403] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4: Time spent bootstrapping tablet: real 0.051s user 0.014s sys 0.008s
I20251025 20:03:11.813849 30287 raft_consensus.cc:359] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.817543 30312 raft_consensus.cc:515] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.818176 30287 raft_consensus.cc:385] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.818675 30287 raft_consensus.cc:740] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.819756 30312 leader_election.cc:290] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.819561 30287 consensus_queue.cc:260] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.820829 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ce5274ff7fe43fab967b217512f197d" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:11.821698 30245 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ce5274ff7fe43fab967b217512f197d" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7"
W20251025 20:03:11.822185 30062 leader_election.cc:343] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.822299 30245 raft_consensus.cc:3060] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.823066 30304 raft_consensus.cc:493] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.823557 30304 raft_consensus.cc:515] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.824028 30286 tablet_bootstrap.cc:492] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7: No bootstrap required, opened a new log
I20251025 20:03:11.824658 30286 ts_tablet_manager.cc:1403] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7: Time spent bootstrapping tablet: real 0.056s user 0.020s sys 0.010s
I20251025 20:03:11.825604 30304 leader_election.cc:290] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.826762 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:11.827642 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
I20251025 20:03:11.828428 30097 raft_consensus.cc:2393] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17d90a5990d3424db25cf46b32327df5 in current term 1: Already voted for candidate d00e76da60c94a3482243bba69caeee7 in this term.
I20251025 20:03:11.828161 30286 raft_consensus.cc:359] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:11.828940 30286 raft_consensus.cc:385] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.829322 30286 raft_consensus.cc:740] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.830118 30135 leader_election.cc:304] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5; no voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7
I20251025 20:03:11.830981 30304 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.818183 29994 catalog_manager.cc:5649] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d90a5990d3424db25cf46b32327df5 (127.28.192.194). New cstate: current_term: 1 leader_uuid: "17d90a5990d3424db25cf46b32327df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:11.834913 30287 ts_tablet_manager.cc:1434] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4: Time spent starting tablet: real 0.033s user 0.005s sys 0.004s
I20251025 20:03:11.836169 30286 consensus_queue.cc:260] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:11.837657 30245 raft_consensus.cc:2468] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 1.
I20251025 20:03:11.838636 30286 ts_tablet_manager.cc:1434] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7: Time spent starting tablet: real 0.014s user 0.006s sys 0.004s
I20251025 20:03:11.843084 30286 tablet_bootstrap.cc:492] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7: Bootstrap starting.
I20251025 20:03:11.847743 30304 raft_consensus.cc:2749] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:11.847929 30060 leader_election.cc:304] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters: 17d90a5990d3424db25cf46b32327df5
I20251025 20:03:11.849177 30296 raft_consensus.cc:2804] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:11.842438 30312 raft_consensus.cc:493] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.850891 30312 raft_consensus.cc:515] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.852847 30287 tablet_bootstrap.cc:492] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4: Bootstrap starting.
I20251025 20:03:11.857170 30312 leader_election.cc:290] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.858358 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:11.859313 30246 raft_consensus.cc:2468] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 0.
I20251025 20:03:11.859812 30286 tablet_bootstrap.cc:654] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.860355 30285 tablet_bootstrap.cc:492] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5: No bootstrap required, opened a new log
I20251025 20:03:11.861929 30285 ts_tablet_manager.cc:1403] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5: Time spent bootstrapping tablet: real 0.103s user 0.014s sys 0.024s
I20251025 20:03:11.862114 30060 leader_election.cc:304] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:11.862540 30287 tablet_bootstrap.cc:654] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.864882 30312 raft_consensus.cc:2804] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:11.864960 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5" is_pre_election: true
I20251025 20:03:11.865653 30312 raft_consensus.cc:493] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:11.866218 30312 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.871811 30285 raft_consensus.cc:359] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.873035 30285 raft_consensus.cc:385] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.873487 30285 raft_consensus.cc:740] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Initialized, Role: FOLLOWER
W20251025 20:03:11.874344 30062 leader_election.cc:343] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.876287 30292 raft_consensus.cc:493] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.877096 30292 raft_consensus.cc:515] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.882494 30292 leader_election.cc:290] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617)
I20251025 20:03:11.883421 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea6745d0705144b0999e3a2f7a35e8da" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
I20251025 20:03:11.884007 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea6745d0705144b0999e3a2f7a35e8da" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5" is_pre_election: true
I20251025 20:03:11.884950 30172 raft_consensus.cc:2468] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 0.
W20251025 20:03:11.886055 30209 leader_election.cc:343] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.887387 30211 leader_election.cc:304] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters: be90690510254d95a22a7def29abc3c4
I20251025 20:03:11.888808 30296 raft_consensus.cc:697] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:11.889336 30292 raft_consensus.cc:2804] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:11.889961 30292 raft_consensus.cc:493] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:11.890539 30292 raft_consensus.cc:3060] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.890213 30296 consensus_queue.cc:237] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.894233 30285 consensus_queue.cc:260] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.905172 30286 tablet_bootstrap.cc:492] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7: No bootstrap required, opened a new log
I20251025 20:03:11.905740 30286 ts_tablet_manager.cc:1403] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7: Time spent bootstrapping tablet: real 0.063s user 0.031s sys 0.019s
I20251025 20:03:11.908905 30286 raft_consensus.cc:359] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.910010 30304 raft_consensus.cc:493] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.910338 30292 raft_consensus.cc:515] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.910635 30304 raft_consensus.cc:515] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.911840 30287 tablet_bootstrap.cc:492] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4: No bootstrap required, opened a new log
I20251025 20:03:11.912381 30287 ts_tablet_manager.cc:1403] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4: Time spent bootstrapping tablet: real 0.060s user 0.017s sys 0.031s
I20251025 20:03:11.912699 30292 leader_election.cc:290] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617)
I20251025 20:03:11.913364 30285 ts_tablet_manager.cc:1434] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5: Time spent starting tablet: real 0.051s user 0.010s sys 0.002s
I20251025 20:03:11.914557 30285 tablet_bootstrap.cc:492] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5: Bootstrap starting.
I20251025 20:03:11.915002 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
I20251025 20:03:11.915844 30287 raft_consensus.cc:359] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.916587 30287 raft_consensus.cc:385] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.916893 30287 raft_consensus.cc:740] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.917060 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea6745d0705144b0999e3a2f7a35e8da" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:11.909767 30286 raft_consensus.cc:385] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.917830 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea6745d0705144b0999e3a2f7a35e8da" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:11.918159 30286 raft_consensus.cc:740] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.918463 30172 raft_consensus.cc:3060] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.904645 30312 raft_consensus.cc:515] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
W20251025 20:03:11.918771 30209 leader_election.cc:343] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:11.919400 30096 raft_consensus.cc:2393] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17d90a5990d3424db25cf46b32327df5 in current term 1: Already voted for candidate be90690510254d95a22a7def29abc3c4 in this term.
I20251025 20:03:11.917637 30287 consensus_queue.cc:260] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.922168 30287 ts_tablet_manager.cc:1434] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20251025 20:03:11.922255 30312 leader_election.cc:290] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.924875 30287 tablet_bootstrap.cc:492] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4: Bootstrap starting.
I20251025 20:03:11.925057 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:11.918951 30286 consensus_queue.cc:260] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.927531 30304 leader_election.cc:290] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.925819 30171 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.928755 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:11.929392 30246 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.931386 30245 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:11.933063 30135 leader_election.cc:304] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5; no voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7
I20251025 20:03:11.933703 30287 tablet_bootstrap.cc:654] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.936229 30286 ts_tablet_manager.cc:1434] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20251025 20:03:11.943894 30246 raft_consensus.cc:2468] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 1.
I20251025 20:03:11.944293 30285 tablet_bootstrap.cc:654] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.945444 30060 leader_election.cc:304] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:11.946640 30296 raft_consensus.cc:2804] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:11.947276 29994 catalog_manager.cc:5649] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:11.947364 30296 raft_consensus.cc:697] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:11.950778 30296 consensus_queue.cc:237] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.943308 30171 raft_consensus.cc:2468] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 1.
I20251025 20:03:11.949803 30172 raft_consensus.cc:2468] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 1.
I20251025 20:03:11.956095 30211 leader_election.cc:304] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters: be90690510254d95a22a7def29abc3c4
I20251025 20:03:11.958334 30292 raft_consensus.cc:2804] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:11.959085 30292 raft_consensus.cc:697] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 1 LEADER]: Becoming Leader. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:11.960438 30292 consensus_queue.cc:237] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.965747 30285 tablet_bootstrap.cc:492] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5: No bootstrap required, opened a new log
I20251025 20:03:11.966588 30285 ts_tablet_manager.cc:1403] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.005s
I20251025 20:03:11.968050 30296 raft_consensus.cc:493] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:11.968605 30287 tablet_bootstrap.cc:492] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4: No bootstrap required, opened a new log
I20251025 20:03:11.968577 30296 raft_consensus.cc:515] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:11.969197 30287 ts_tablet_manager.cc:1403] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.021s
I20251025 20:03:11.969528 30285 raft_consensus.cc:359] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.970316 30285 raft_consensus.cc:385] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.970597 30285 raft_consensus.cc:740] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.970855 30296 leader_election.cc:290] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:11.971294 30285 consensus_queue.cc:260] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.971873 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5" is_pre_election: true
I20251025 20:03:11.972579 30172 raft_consensus.cc:2468] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 0.
I20251025 20:03:11.972404 30287 raft_consensus.cc:359] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.972530 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:11.973208 30287 raft_consensus.cc:385] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:11.973436 30246 raft_consensus.cc:2468] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 0.
I20251025 20:03:11.973587 30287 raft_consensus.cc:740] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Initialized, Role: FOLLOWER
I20251025 20:03:11.974143 30062 leader_election.cc:304] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, be90690510254d95a22a7def29abc3c4; no voters:
I20251025 20:03:11.974434 30287 consensus_queue.cc:260] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:11.975037 30296 raft_consensus.cc:2804] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:11.975481 30296 raft_consensus.cc:493] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:11.975854 30296 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:11.977209 30285 ts_tablet_manager.cc:1434] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20251025 20:03:11.977216 30287 ts_tablet_manager.cc:1434] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251025 20:03:11.978569 30285 tablet_bootstrap.cc:492] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5: Bootstrap starting.
I20251025 20:03:11.978677 30287 tablet_bootstrap.cc:492] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4: Bootstrap starting.
I20251025 20:03:11.955729 30304 raft_consensus.cc:2749] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:11.985872 30285 tablet_bootstrap.cc:654] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.985872 30287 tablet_bootstrap.cc:654] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:11.995647 29993 catalog_manager.cc:5649] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:12.001631 30296 raft_consensus.cc:515] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:12.004992 30172 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:12.005760 30172 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.005298 29994 catalog_manager.cc:5649] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d00e76da60c94a3482243bba69caeee7 (127.28.192.195). New cstate: current_term: 1 leader_uuid: "d00e76da60c94a3482243bba69caeee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:12.014634 30296 leader_election.cc:290] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.018342 30285 tablet_bootstrap.cc:492] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5: No bootstrap required, opened a new log
I20251025 20:03:12.018983 30285 ts_tablet_manager.cc:1403] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.024s
I20251025 20:03:12.018949 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:12.019621 30246 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.020160 30287 tablet_bootstrap.cc:492] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4: No bootstrap required, opened a new log
I20251025 20:03:12.020637 30287 ts_tablet_manager.cc:1403] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.021s
I20251025 20:03:12.022051 30285 raft_consensus.cc:359] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.022733 30285 raft_consensus.cc:385] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:12.022992 30285 raft_consensus.cc:740] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Initialized, Role: FOLLOWER
I20251025 20:03:12.023139 30287 raft_consensus.cc:359] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.023793 30287 raft_consensus.cc:385] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:12.024147 30287 raft_consensus.cc:740] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Initialized, Role: FOLLOWER
I20251025 20:03:12.023831 30285 consensus_queue.cc:260] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.024971 30291 raft_consensus.cc:493] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:12.024837 30287 consensus_queue.cc:260] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.025409 30291 raft_consensus.cc:515] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.027500 30287 ts_tablet_manager.cc:1434] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251025 20:03:12.027575 30291 leader_election.cc:290] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.028625 30287 tablet_bootstrap.cc:492] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4: Bootstrap starting.
I20251025 20:03:12.031297 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5961f6062df94219941afc0503ff3033" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
I20251025 20:03:12.032176 30096 raft_consensus.cc:2468] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 0.
I20251025 20:03:12.029855 30245 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5961f6062df94219941afc0503ff3033" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:12.033340 30245 raft_consensus.cc:2468] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 0.
I20251025 20:03:12.034639 30135 leader_election.cc:304] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:12.035502 30287 tablet_bootstrap.cc:654] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:12.037151 30304 raft_consensus.cc:2804] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:12.038554 30304 raft_consensus.cc:493] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:12.039021 30304 raft_consensus.cc:3060] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.031406 30246 raft_consensus.cc:2468] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 1.
I20251025 20:03:12.041024 30060 leader_election.cc:304] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:12.041971 30296 raft_consensus.cc:2804] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:12.042001 30285 ts_tablet_manager.cc:1434] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20251025 20:03:12.042550 30296 raft_consensus.cc:697] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:12.043458 30285 tablet_bootstrap.cc:492] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5: Bootstrap starting.
I20251025 20:03:12.043437 30296 consensus_queue.cc:237] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:12.053076 30285 tablet_bootstrap.cc:654] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:12.058138 30172 raft_consensus.cc:2468] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 1.
I20251025 20:03:12.057531 30304 raft_consensus.cc:515] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.064769 30285 tablet_bootstrap.cc:492] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5: No bootstrap required, opened a new log
I20251025 20:03:12.066099 30304 leader_election.cc:290] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.067445 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5961f6062df94219941afc0503ff3033" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:12.068037 30246 raft_consensus.cc:3060] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.069254 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5961f6062df94219941afc0503ff3033" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:12.069516 30287 tablet_bootstrap.cc:492] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4: No bootstrap required, opened a new log
I20251025 20:03:12.069926 30096 raft_consensus.cc:3060] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.070115 30287 ts_tablet_manager.cc:1403] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.015s
I20251025 20:03:12.069048 29994 catalog_manager.cc:5649] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:12.073065 30287 raft_consensus.cc:359] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.073817 30287 raft_consensus.cc:385] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:12.074134 30287 raft_consensus.cc:740] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Initialized, Role: FOLLOWER
I20251025 20:03:12.074800 30287 consensus_queue.cc:260] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.076042 30246 raft_consensus.cc:2468] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 1.
I20251025 20:03:12.077194 30287 ts_tablet_manager.cc:1434] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251025 20:03:12.077580 30135 leader_election.cc:304] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:12.078203 30096 raft_consensus.cc:2468] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 1.
I20251025 20:03:12.078219 30287 tablet_bootstrap.cc:492] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4: Bootstrap starting.
I20251025 20:03:12.078483 30304 raft_consensus.cc:2804] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:12.065503 30285 ts_tablet_manager.cc:1403] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.011s
I20251025 20:03:12.080505 30304 raft_consensus.cc:697] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Becoming Leader. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:12.081488 30304 consensus_queue.cc:237] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.083057 30285 raft_consensus.cc:359] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.083822 30285 raft_consensus.cc:385] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:12.084110 30285 raft_consensus.cc:740] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Initialized, Role: FOLLOWER
I20251025 20:03:12.084810 30285 consensus_queue.cc:260] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.085862 30287 tablet_bootstrap.cc:654] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:12.086681 30291 raft_consensus.cc:493] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:12.087311 30285 ts_tablet_manager.cc:1434] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251025 20:03:12.087224 30291 raft_consensus.cc:515] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.089653 30291 leader_election.cc:290] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.093029 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4" is_pre_election: true
I20251025 20:03:12.094091 30096 raft_consensus.cc:2468] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 0.
I20251025 20:03:12.094455 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:12.095315 30246 raft_consensus.cc:2468] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 0.
I20251025 20:03:12.096073 30135 leader_election.cc:304] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, be90690510254d95a22a7def29abc3c4; no voters:
I20251025 20:03:12.097494 30304 raft_consensus.cc:2804] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:12.097958 30304 raft_consensus.cc:493] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:12.098352 30304 raft_consensus.cc:3060] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.099828 30287 tablet_bootstrap.cc:492] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4: No bootstrap required, opened a new log
I20251025 20:03:12.099737 29995 catalog_manager.cc:5649] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d90a5990d3424db25cf46b32327df5 (127.28.192.194). New cstate: current_term: 1 leader_uuid: "17d90a5990d3424db25cf46b32327df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:12.100404 30287 ts_tablet_manager.cc:1403] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.015s
I20251025 20:03:12.103379 30287 raft_consensus.cc:359] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.104100 30287 raft_consensus.cc:385] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:12.104391 30287 raft_consensus.cc:740] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Initialized, Role: FOLLOWER
I20251025 20:03:12.105047 30287 consensus_queue.cc:260] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.106495 30304 raft_consensus.cc:515] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.107636 30287 ts_tablet_manager.cc:1434] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251025 20:03:12.108671 30304 leader_election.cc:290] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.110009 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:12.110677 30096 raft_consensus.cc:3060] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.110556 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:12.111302 30246 raft_consensus.cc:3060] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.120383 30246 raft_consensus.cc:2468] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 1.
I20251025 20:03:12.121241 30296 consensus_queue.cc:1048] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.121655 30135 leader_election.cc:304] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:12.122515 30304 raft_consensus.cc:2804] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:12.122685 30096 raft_consensus.cc:2468] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 1.
I20251025 20:03:12.123063 30304 raft_consensus.cc:697] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Becoming Leader. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:12.124046 30304 consensus_queue.cc:237] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.130228 30314 consensus_queue.cc:1048] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.138983 29995 catalog_manager.cc:5649] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d90a5990d3424db25cf46b32327df5 (127.28.192.194). New cstate: current_term: 1 leader_uuid: "17d90a5990d3424db25cf46b32327df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:12.145311 30314 consensus_queue.cc:1048] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.148797 30329 raft_consensus.cc:493] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:12.149389 30329 raft_consensus.cc:515] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.156454 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38e64303be304c388287465e021ab853" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5" is_pre_election: true
I20251025 20:03:12.157524 30329 leader_election.cc:290] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.161747 30245 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38e64303be304c388287465e021ab853" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:12.163319 30245 raft_consensus.cc:2393] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be90690510254d95a22a7def29abc3c4 in current term 1: Already voted for candidate 17d90a5990d3424db25cf46b32327df5 in this term.
I20251025 20:03:12.164659 30171 raft_consensus.cc:3060] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.174715 30329 consensus_queue.cc:1048] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.164532 30060 leader_election.cc:304] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be90690510254d95a22a7def29abc3c4; no voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7
I20251025 20:03:12.182099 30312 raft_consensus.cc:3060] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.181855 30291 consensus_queue.cc:1048] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.195672 30312 raft_consensus.cc:2749] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:12.216406 30096 raft_consensus.cc:3060] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.226540 30321 consensus_queue.cc:1048] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.248512 30246 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ea6745d0705144b0999e3a2f7a35e8da"
dest_uuid: "d00e76da60c94a3482243bba69caeee7"
mode: GRACEFUL
new_leader_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:58218
I20251025 20:03:12.249172 30246 raft_consensus.cc:606] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 1 LEADER]: Received request to transfer leadership to TS be90690510254d95a22a7def29abc3c4
I20251025 20:03:12.263913 30170 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "38e64303be304c388287465e021ab853"
dest_uuid: "17d90a5990d3424db25cf46b32327df5"
mode: GRACEFUL
new_leader_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:41436
I20251025 20:03:12.264554 30170 raft_consensus.cc:606] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Received request to transfer leadership to TS be90690510254d95a22a7def29abc3c4
I20251025 20:03:12.269977 30170 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2038c9436df6436d92f7a25bfc5474bf"
dest_uuid: "17d90a5990d3424db25cf46b32327df5"
mode: GRACEFUL
new_leader_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:41436
I20251025 20:03:12.270555 30170 raft_consensus.cc:606] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Received request to transfer leadership to TS be90690510254d95a22a7def29abc3c4
I20251025 20:03:12.271970 30312 raft_consensus.cc:493] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:12.272469 30312 raft_consensus.cc:515] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.274745 30312 leader_election.cc:290] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.275177 30170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea6745d0705144b0999e3a2f7a35e8da" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d90a5990d3424db25cf46b32327df5" is_pre_election: true
I20251025 20:03:12.275372 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea6745d0705144b0999e3a2f7a35e8da" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e76da60c94a3482243bba69caeee7" is_pre_election: true
I20251025 20:03:12.276139 30170 raft_consensus.cc:2393] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be90690510254d95a22a7def29abc3c4 in current term 1: Already voted for candidate d00e76da60c94a3482243bba69caeee7 in this term.
I20251025 20:03:12.277093 30062 leader_election.cc:304] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be90690510254d95a22a7def29abc3c4; no voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7
I20251025 20:03:12.278174 30312 raft_consensus.cc:3060] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.283102 30246 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b"
dest_uuid: "d00e76da60c94a3482243bba69caeee7"
mode: GRACEFUL
new_leader_uuid: "17d90a5990d3424db25cf46b32327df5"
from {username='slave'} at 127.0.0.1:58218
I20251025 20:03:12.283696 30246 raft_consensus.cc:606] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 1 LEADER]: Received request to transfer leadership to TS 17d90a5990d3424db25cf46b32327df5
I20251025 20:03:12.284556 30312 raft_consensus.cc:2749] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:12.301183 30096 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4128ceefc2c541ccb009b58235c071e6"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
mode: GRACEFUL
new_leader_uuid: "17d90a5990d3424db25cf46b32327df5"
from {username='slave'} at 127.0.0.1:55432
I20251025 20:03:12.301836 30096 raft_consensus.cc:606] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Received request to transfer leadership to TS 17d90a5990d3424db25cf46b32327df5
I20251025 20:03:12.307049 30096 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7ce5274ff7fe43fab967b217512f197d"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
mode: GRACEFUL
new_leader_uuid: "d00e76da60c94a3482243bba69caeee7"
from {username='slave'} at 127.0.0.1:55432
I20251025 20:03:12.307662 30096 raft_consensus.cc:606] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Received request to transfer leadership to TS d00e76da60c94a3482243bba69caeee7
I20251025 20:03:12.325321 30321 consensus_queue.cc:1048] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:12.337603 30291 consensus_queue.cc:1048] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:12.346810 30304 raft_consensus.cc:993] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5: : Instructing follower be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:12.347275 30291 raft_consensus.cc:1081] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Signalling peer be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:12.348776 30096 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "38e64303be304c388287465e021ab853"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:55422
I20251025 20:03:12.349435 30096 raft_consensus.cc:493] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:12.349824 30096 raft_consensus.cc:3060] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.354934 30312 consensus_queue.cc:1048] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.357039 30096 raft_consensus.cc:515] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.359335 30096 leader_election.cc:290] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 2 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.360213 30170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38e64303be304c388287465e021ab853" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:12.360935 30170 raft_consensus.cc:3055] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:12.361289 30170 raft_consensus.cc:740] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:12.361799 30245 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38e64303be304c388287465e021ab853" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:12.362442 30245 raft_consensus.cc:3060] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.362279 30170 consensus_queue.cc:260] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.363711 30170 raft_consensus.cc:3060] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.365777 30329 raft_consensus.cc:993] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4: : Instructing follower d00e76da60c94a3482243bba69caeee7 to start an election
I20251025 20:03:12.365810 30292 consensus_queue.cc:1048] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.366336 30329 raft_consensus.cc:1081] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Signalling peer d00e76da60c94a3482243bba69caeee7 to start an election
I20251025 20:03:12.368772 30244 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "7ce5274ff7fe43fab967b217512f197d"
dest_uuid: "d00e76da60c94a3482243bba69caeee7"
from {username='slave'} at 127.0.0.1:58198
I20251025 20:03:12.369418 30244 raft_consensus.cc:493] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:12.369817 30244 raft_consensus.cc:3060] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.374495 30171 raft_consensus.cc:3060] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:12.374895 30245 raft_consensus.cc:2468] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 2.
I20251025 20:03:12.376637 30060 leader_election.cc:304] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:12.378243 30296 raft_consensus.cc:2804] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:12.378976 30296 raft_consensus.cc:697] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:12.380205 30296 consensus_queue.cc:237] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.384079 30170 raft_consensus.cc:2468] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 2.
I20251025 20:03:12.386739 30244 raft_consensus.cc:515] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.389744 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ce5274ff7fe43fab967b217512f197d" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:12.390365 30296 consensus_queue.cc:1048] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.393386 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ce5274ff7fe43fab967b217512f197d" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:12.393878 30097 raft_consensus.cc:3055] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:12.394340 30097 raft_consensus.cc:740] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:12.395206 30097 consensus_queue.cc:260] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.396858 30246 raft_consensus.cc:1240] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Rejecting Update request from peer be90690510254d95a22a7def29abc3c4 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:12.396381 30097 raft_consensus.cc:3060] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.399267 30292 raft_consensus.cc:993] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7: : Instructing follower be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:12.392223 30244 leader_election.cc:290] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 2 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617)
I20251025 20:03:12.399730 30292 raft_consensus.cc:1081] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 1 LEADER]: Signalling peer be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:12.409092 30097 raft_consensus.cc:2468] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 2.
I20251025 20:03:12.413796 30314 consensus_queue.cc:1048] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251025 20:03:12.417088 30097 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ea6745d0705144b0999e3a2f7a35e8da"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:55410
I20251025 20:03:12.417991 30097 raft_consensus.cc:493] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:12.418414 30097 raft_consensus.cc:3060] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.421444 29993 catalog_manager.cc:5649] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 1 to 2, leader changed from 17d90a5990d3424db25cf46b32327df5 (127.28.192.194) to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:12.423565 30209 leader_election.cc:304] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters: 17d90a5990d3424db25cf46b32327df5
I20251025 20:03:12.424654 30292 raft_consensus.cc:2804] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:12.425141 30292 raft_consensus.cc:697] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Becoming Leader. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:12.425952 30292 consensus_queue.cc:237] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.426621 30097 raft_consensus.cc:515] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.429581 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea6745d0705144b0999e3a2f7a35e8da" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:12.430158 30097 leader_election.cc:290] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 2 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.432246 30244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea6745d0705144b0999e3a2f7a35e8da" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:12.432770 30244 raft_consensus.cc:3055] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:12.433022 30244 raft_consensus.cc:740] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:12.433781 30244 consensus_queue.cc:260] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.434963 30244 raft_consensus.cc:3060] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.442276 30244 raft_consensus.cc:2468] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 2.
I20251025 20:03:12.441844 29994 catalog_manager.cc:5649] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 reported cstate change: term changed from 1 to 2, leader changed from be90690510254d95a22a7def29abc3c4 (127.28.192.193) to d00e76da60c94a3482243bba69caeee7 (127.28.192.195). New cstate: current_term: 2 leader_uuid: "d00e76da60c94a3482243bba69caeee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:12.443517 30060 leader_election.cc:304] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters: 17d90a5990d3424db25cf46b32327df5
I20251025 20:03:12.444423 30296 raft_consensus.cc:2804] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:12.444931 30296 raft_consensus.cc:697] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:12.445827 30296 consensus_queue.cc:237] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.452270 30296 consensus_queue.cc:1048] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:12.455693 29995 catalog_manager.cc:5649] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 1 to 2, leader changed from d00e76da60c94a3482243bba69caeee7 (127.28.192.195) to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:12.469372 30296 consensus_queue.cc:1048] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.490039 30291 consensus_queue.cc:1048] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.502238 30291 consensus_queue.cc:1048] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:12.523321 30312 consensus_queue.cc:1048] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:12.524211 30296 consensus_queue.cc:1048] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:12.589614 30291 consensus_queue.cc:1048] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:12.603579 30291 consensus_queue.cc:1048] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:12.685935 30329 raft_consensus.cc:993] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4: : Instructing follower 17d90a5990d3424db25cf46b32327df5 to start an election
I20251025 20:03:12.686420 30296 raft_consensus.cc:1081] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Signalling peer 17d90a5990d3424db25cf46b32327df5 to start an election
I20251025 20:03:12.688153 30171 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4128ceefc2c541ccb009b58235c071e6"
dest_uuid: "17d90a5990d3424db25cf46b32327df5"
from {username='slave'} at 127.0.0.1:41428
I20251025 20:03:12.688692 30171 raft_consensus.cc:493] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:12.689000 30171 raft_consensus.cc:3060] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.694306 30171 raft_consensus.cc:515] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.696223 30171 leader_election.cc:290] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 2 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.697221 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4128ceefc2c541ccb009b58235c071e6" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:12.697955 30244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4128ceefc2c541ccb009b58235c071e6" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:12.698813 30097 raft_consensus.cc:3055] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:12.699126 30244 raft_consensus.cc:3060] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.700067 30304 raft_consensus.cc:993] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5: : Instructing follower be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:12.700539 30304 raft_consensus.cc:1081] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Signalling peer be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:12.702351 30096 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2038c9436df6436d92f7a25bfc5474bf"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:55422
I20251025 20:03:12.702973 30096 raft_consensus.cc:493] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:12.703348 30096 raft_consensus.cc:3060] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.699417 30097 raft_consensus.cc:740] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:12.704914 30097 consensus_queue.cc:260] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.706586 30097 raft_consensus.cc:3060] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.707405 30244 raft_consensus.cc:2468] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 2.
I20251025 20:03:12.708688 30135 leader_election.cc:304] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:12.709444 30373 raft_consensus.cc:2804] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:12.709861 30373 raft_consensus.cc:697] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 2 LEADER]: Becoming Leader. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:12.710795 30373 consensus_queue.cc:237] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.711549 30096 raft_consensus.cc:515] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.714691 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2038c9436df6436d92f7a25bfc5474bf" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:12.715114 30096 leader_election.cc:290] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 2 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.715472 30171 raft_consensus.cc:3055] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:12.715631 30244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2038c9436df6436d92f7a25bfc5474bf" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:12.714764 30097 raft_consensus.cc:2468] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 2.
I20251025 20:03:12.715870 30171 raft_consensus.cc:740] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:12.716279 30244 raft_consensus.cc:3060] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.716873 30171 consensus_queue.cc:260] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.718291 30171 raft_consensus.cc:3060] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.720027 29995 catalog_manager.cc:5649] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 reported cstate change: term changed from 1 to 2, leader changed from be90690510254d95a22a7def29abc3c4 (127.28.192.193) to 17d90a5990d3424db25cf46b32327df5 (127.28.192.194). New cstate: current_term: 2 leader_uuid: "17d90a5990d3424db25cf46b32327df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:12.722649 30244 raft_consensus.cc:2468] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 2.
I20251025 20:03:12.723994 30060 leader_election.cc:304] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:12.724933 30296 raft_consensus.cc:2804] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:12.725448 30296 raft_consensus.cc:697] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:12.726318 30171 raft_consensus.cc:2468] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 2.
I20251025 20:03:12.726409 30296 consensus_queue.cc:237] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.734369 29993 catalog_manager.cc:5649] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 1 to 2, leader changed from 17d90a5990d3424db25cf46b32327df5 (127.28.192.194) to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:12.738096 30314 raft_consensus.cc:993] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7: : Instructing follower 17d90a5990d3424db25cf46b32327df5 to start an election
I20251025 20:03:12.738595 30292 raft_consensus.cc:1081] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 1 LEADER]: Signalling peer 17d90a5990d3424db25cf46b32327df5 to start an election
I20251025 20:03:12.740095 30171 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b"
dest_uuid: "17d90a5990d3424db25cf46b32327df5"
from {username='slave'} at 127.0.0.1:41414
I20251025 20:03:12.740643 30171 raft_consensus.cc:493] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:12.740928 30171 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.747171 30171 raft_consensus.cc:515] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.748951 30171 leader_election.cc:290] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 2 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:12.749845 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:12.750114 30244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:12.750558 30097 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.750658 30244 raft_consensus.cc:3055] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:12.751020 30244 raft_consensus.cc:740] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:12.751881 30244 consensus_queue.cc:260] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.753198 30244 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.758657 30097 raft_consensus.cc:2468] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 2.
I20251025 20:03:12.759938 30135 leader_election.cc:304] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, be90690510254d95a22a7def29abc3c4; no voters:
I20251025 20:03:12.760643 30373 raft_consensus.cc:2804] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:12.760617 30244 raft_consensus.cc:2468] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 2.
I20251025 20:03:12.761221 30373 raft_consensus.cc:697] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 2 LEADER]: Becoming Leader. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:12.762176 30373 consensus_queue.cc:237] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:12.769169 29993 catalog_manager.cc:5649] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 reported cstate change: term changed from 1 to 2, leader changed from d00e76da60c94a3482243bba69caeee7 (127.28.192.195) to 17d90a5990d3424db25cf46b32327df5 (127.28.192.194). New cstate: current_term: 2 leader_uuid: "17d90a5990d3424db25cf46b32327df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:12.794422 30171 raft_consensus.cc:1275] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:12.795768 30296 consensus_queue.cc:1048] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:12.805505 30244 raft_consensus.cc:1275] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:12.807211 30329 consensus_queue.cc:1048] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:12.880446 30097 raft_consensus.cc:1275] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Refusing update from remote peer d00e76da60c94a3482243bba69caeee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:12.881822 30292 consensus_queue.cc:1048] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:12.892172 30171 raft_consensus.cc:3060] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:12.900787 30171 raft_consensus.cc:1275] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Refusing update from remote peer d00e76da60c94a3482243bba69caeee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:12.902127 30292 consensus_queue.cc:1048] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:13.031560 30244 raft_consensus.cc:1275] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:13.033273 30329 consensus_queue.cc:1048] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:13.045197 30171 raft_consensus.cc:3060] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:13.053687 30171 raft_consensus.cc:1275] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:13.055156 30312 consensus_queue.cc:1048] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:13.145395 30097 raft_consensus.cc:1275] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Refusing update from remote peer 17d90a5990d3424db25cf46b32327df5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:13.147167 30304 consensus_queue.cc:1048] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:13.155078 30244 raft_consensus.cc:1275] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Refusing update from remote peer 17d90a5990d3424db25cf46b32327df5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:13.157193 30373 consensus_queue.cc:1048] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:13.228734 30171 raft_consensus.cc:1275] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:13.230436 30312 consensus_queue.cc:1048] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:13.240178 30244 raft_consensus.cc:1275] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:13.242875 30296 consensus_queue.cc:1048] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:13.285018 30097 raft_consensus.cc:1275] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Refusing update from remote peer 17d90a5990d3424db25cf46b32327df5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:13.286377 30373 consensus_queue.cc:1048] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:13.295866 30246 raft_consensus.cc:1275] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Refusing update from remote peer 17d90a5990d3424db25cf46b32327df5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:13.298182 30291 consensus_queue.cc:1048] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:15.338289 29443 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20251025 20:03:15.338985 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:15.357470 30097 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
mode: GRACEFUL
new_leader_uuid: "d00e76da60c94a3482243bba69caeee7"
from {username='slave'} at 127.0.0.1:55446
I20251025 20:03:15.358088 30097 raft_consensus.cc:606] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Received request to transfer leadership to TS d00e76da60c94a3482243bba69caeee7
I20251025 20:03:15.360440 30097 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
mode: GRACEFUL
new_leader_uuid: "d00e76da60c94a3482243bba69caeee7"
from {username='slave'} at 127.0.0.1:55446
I20251025 20:03:15.360958 30097 raft_consensus.cc:606] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Received request to transfer leadership to TS d00e76da60c94a3482243bba69caeee7
I20251025 20:03:15.361840 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251025 20:03:15.362329 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:15.864513 30329 raft_consensus.cc:993] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4: : Instructing follower d00e76da60c94a3482243bba69caeee7 to start an election
I20251025 20:03:15.865051 30384 raft_consensus.cc:1081] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Signalling peer d00e76da60c94a3482243bba69caeee7 to start an election
I20251025 20:03:15.866586 30244 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8"
dest_uuid: "d00e76da60c94a3482243bba69caeee7"
from {username='slave'} at 127.0.0.1:58198
I20251025 20:03:15.867053 30244 raft_consensus.cc:493] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:15.867307 30244 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:15.872699 30244 raft_consensus.cc:515] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:15.874389 30244 leader_election.cc:290] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 2 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617)
I20251025 20:03:15.875385 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:15.875766 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:15.875958 30097 raft_consensus.cc:3055] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:15.876338 30171 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:15.876364 30097 raft_consensus.cc:740] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:15.877219 30097 consensus_queue.cc:260] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:15.878259 30097 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:15.882830 30171 raft_consensus.cc:2468] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 2.
I20251025 20:03:15.884230 30211 leader_election.cc:304] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:15.884529 30097 raft_consensus.cc:2468] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 2.
I20251025 20:03:15.885116 30292 raft_consensus.cc:2804] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:15.886094 30292 raft_consensus.cc:697] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Becoming Leader. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:15.886911 30292 consensus_queue.cc:237] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:15.896013 29993 catalog_manager.cc:5649] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 reported cstate change: term changed from 1 to 2, leader changed from be90690510254d95a22a7def29abc3c4 (127.28.192.193) to d00e76da60c94a3482243bba69caeee7 (127.28.192.195). New cstate: current_term: 2 leader_uuid: "d00e76da60c94a3482243bba69caeee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:15.920378 30329 raft_consensus.cc:993] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4: : Instructing follower d00e76da60c94a3482243bba69caeee7 to start an election
I20251025 20:03:15.920850 30391 raft_consensus.cc:1081] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Signalling peer d00e76da60c94a3482243bba69caeee7 to start an election
I20251025 20:03:15.922418 30244 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7"
dest_uuid: "d00e76da60c94a3482243bba69caeee7"
from {username='slave'} at 127.0.0.1:58198
I20251025 20:03:15.922940 30244 raft_consensus.cc:493] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:15.923224 30244 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:15.928601 30244 raft_consensus.cc:515] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:15.930338 30244 leader_election.cc:290] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 2 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415)
I20251025 20:03:15.931432 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:15.931437 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:15.932147 30171 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:15.932257 30097 raft_consensus.cc:3055] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:15.932591 30097 raft_consensus.cc:740] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:15.933302 30097 consensus_queue.cc:260] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:15.934715 30097 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:15.938506 30171 raft_consensus.cc:2468] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 2.
I20251025 20:03:15.939612 30211 leader_election.cc:304] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:15.940521 30292 raft_consensus.cc:2804] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:15.940527 30097 raft_consensus.cc:2468] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 2.
I20251025 20:03:15.940996 30292 raft_consensus.cc:697] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Becoming Leader. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:15.942776 30292 consensus_queue.cc:237] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:15.950066 29993 catalog_manager.cc:5649] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 reported cstate change: term changed from 1 to 2, leader changed from be90690510254d95a22a7def29abc3c4 (127.28.192.193) to d00e76da60c94a3482243bba69caeee7 (127.28.192.195). New cstate: current_term: 2 leader_uuid: "d00e76da60c94a3482243bba69caeee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:16.363292 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:16.369221 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:16.369652 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:16.403539 30096 raft_consensus.cc:1275] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Refusing update from remote peer d00e76da60c94a3482243bba69caeee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:16.404789 30292 consensus_queue.cc:1048] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:16.416611 30171 raft_consensus.cc:1275] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Refusing update from remote peer d00e76da60c94a3482243bba69caeee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:16.418267 30385 consensus_queue.cc:1048] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:16.500752 30171 raft_consensus.cc:1275] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Refusing update from remote peer d00e76da60c94a3482243bba69caeee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:16.502494 30292 consensus_queue.cc:1048] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:16.511880 30096 raft_consensus.cc:1275] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Refusing update from remote peer d00e76da60c94a3482243bba69caeee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:16.513674 30385 consensus_queue.cc:1048] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:17.371171 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:17.378975 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:17.379489 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:18.380694 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:18.386570 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:18.387059 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:19.387933 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:19.393327 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:19.393862 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:20.394908 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:20.402261 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:20.402738 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:21.403766 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:21.410524 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:21.410943 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:22.411777 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:22.417148 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:22.417562 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:23.418588 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:23.424083 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:23.424526 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:24.425498 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:24.431023 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:24.431524 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:25.462374 29443 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20251025 20:03:25.467471 30246 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7"
dest_uuid: "d00e76da60c94a3482243bba69caeee7"
mode: GRACEFUL
new_leader_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:58218
I20251025 20:03:25.468151 30246 raft_consensus.cc:606] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Received request to transfer leadership to TS be90690510254d95a22a7def29abc3c4
I20251025 20:03:25.479305 30246 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8"
dest_uuid: "d00e76da60c94a3482243bba69caeee7"
mode: GRACEFUL
new_leader_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:58218
I20251025 20:03:25.479828 30246 raft_consensus.cc:606] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Received request to transfer leadership to TS be90690510254d95a22a7def29abc3c4
I20251025 20:03:25.484597 30171 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528"
dest_uuid: "17d90a5990d3424db25cf46b32327df5"
mode: GRACEFUL
new_leader_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:41436
I20251025 20:03:25.485150 30171 raft_consensus.cc:606] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Received request to transfer leadership to TS be90690510254d95a22a7def29abc3c4
I20251025 20:03:25.489883 30171 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b"
dest_uuid: "17d90a5990d3424db25cf46b32327df5"
mode: GRACEFUL
new_leader_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:41436
I20251025 20:03:25.490374 30171 raft_consensus.cc:606] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 2 LEADER]: Received request to transfer leadership to TS be90690510254d95a22a7def29abc3c4
I20251025 20:03:25.494782 30171 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5961f6062df94219941afc0503ff3033"
dest_uuid: "17d90a5990d3424db25cf46b32327df5"
mode: GRACEFUL
new_leader_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:41436
I20251025 20:03:25.495277 30171 raft_consensus.cc:606] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Received request to transfer leadership to TS be90690510254d95a22a7def29abc3c4
I20251025 20:03:25.505080 30292 raft_consensus.cc:993] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7: : Instructing follower be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:25.505692 30411 raft_consensus.cc:1081] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Signalling peer be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:25.507268 30096 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:55410
I20251025 20:03:25.507772 30096 raft_consensus.cc:493] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:25.508059 30096 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:25.513723 30096 raft_consensus.cc:515] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.515331 30096 leader_election.cc:290] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 3 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:25.516417 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:25.516417 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:25.517316 30171 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:25.517370 30246 raft_consensus.cc:3055] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:03:25.517786 30246 raft_consensus.cc:740] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:25.518659 30246 consensus_queue.cc:260] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.519986 30246 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:25.525382 30171 raft_consensus.cc:2468] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 3.
I20251025 20:03:25.526806 30062 leader_election.cc:304] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, be90690510254d95a22a7def29abc3c4; no voters:
I20251025 20:03:25.527662 30246 raft_consensus.cc:2468] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 3.
I20251025 20:03:25.527804 30391 raft_consensus.cc:2804] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:25.529704 30391 raft_consensus.cc:697] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:25.530659 30391 consensus_queue.cc:237] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.538147 29995 catalog_manager.cc:5649] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 2 to 3, leader changed from d00e76da60c94a3482243bba69caeee7 (127.28.192.195) to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 3 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:25.567792 30418 raft_consensus.cc:993] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5: : Instructing follower be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:25.568526 30406 raft_consensus.cc:1081] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Signalling peer be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:25.569952 30096 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5961f6062df94219941afc0503ff3033"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:55422
I20251025 20:03:25.570482 30096 raft_consensus.cc:493] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:25.570780 30096 raft_consensus.cc:3060] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:25.576174 30096 raft_consensus.cc:515] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.577981 30096 leader_election.cc:290] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 2 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:25.578671 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5961f6062df94219941afc0503ff3033" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:25.578877 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5961f6062df94219941afc0503ff3033" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:25.579267 30246 raft_consensus.cc:3060] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:25.579413 30171 raft_consensus.cc:3055] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:25.579773 30171 raft_consensus.cc:740] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:25.580508 30171 consensus_queue.cc:260] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.581851 30171 raft_consensus.cc:3060] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:25.584807 30246 raft_consensus.cc:2468] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 2.
I20251025 20:03:25.586485 30060 leader_election.cc:304] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:25.587455 30412 raft_consensus.cc:2804] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:25.587585 30171 raft_consensus.cc:2468] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 2.
I20251025 20:03:25.587977 30412 raft_consensus.cc:697] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:25.588965 30412 consensus_queue.cc:237] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.593910 30292 raft_consensus.cc:993] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7: : Instructing follower be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:25.594461 30411 raft_consensus.cc:1081] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Signalling peer be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:25.596123 30096 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:55410
I20251025 20:03:25.596676 30096 raft_consensus.cc:493] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:25.597080 30096 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:25.598372 29995 catalog_manager.cc:5649] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 1 to 2, leader changed from 17d90a5990d3424db25cf46b32327df5 (127.28.192.194) to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:25.603490 30096 raft_consensus.cc:515] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:25.605321 30096 leader_election.cc:290] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 3 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:25.606222 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:25.606680 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:25.607239 30246 raft_consensus.cc:3055] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:03:25.606839 30171 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:25.607522 30246 raft_consensus.cc:740] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:25.608371 30246 consensus_queue.cc:260] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:25.609236 30246 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:25.612367 30171 raft_consensus.cc:2468] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 3.
I20251025 20:03:25.613570 30062 leader_election.cc:304] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, be90690510254d95a22a7def29abc3c4; no voters:
I20251025 20:03:25.614326 30412 raft_consensus.cc:2804] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:25.614718 30412 raft_consensus.cc:697] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:25.615372 30246 raft_consensus.cc:2468] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 3.
I20251025 20:03:25.616068 30412 consensus_queue.cc:237] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:25.623459 29995 catalog_manager.cc:5649] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 2 to 3, leader changed from d00e76da60c94a3482243bba69caeee7 (127.28.192.195) to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 3 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:25.709167 30418 raft_consensus.cc:993] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5: : Instructing follower be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:25.709620 30406 raft_consensus.cc:1081] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 2 LEADER]: Signalling peer be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:25.710973 30096 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:55422
I20251025 20:03:25.711423 30096 raft_consensus.cc:493] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:25.711680 30096 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:25.716384 30096 raft_consensus.cc:515] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.717931 30096 leader_election.cc:290] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 3 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:25.718861 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:25.718988 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:25.719367 30171 raft_consensus.cc:3055] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:03:25.719478 30246 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:25.719640 30171 raft_consensus.cc:740] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:25.720281 30171 consensus_queue.cc:260] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.721220 30171 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:25.724551 30246 raft_consensus.cc:2468] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 3.
I20251025 20:03:25.725476 30060 leader_election.cc:304] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:25.725658 30171 raft_consensus.cc:2468] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 3.
I20251025 20:03:25.726208 30412 raft_consensus.cc:2804] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:25.726620 30412 raft_consensus.cc:697] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:25.727475 30412 consensus_queue.cc:237] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.734325 29995 catalog_manager.cc:5649] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 2 to 3, leader changed from 17d90a5990d3424db25cf46b32327df5 (127.28.192.194) to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 3 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:25.880266 30418 raft_consensus.cc:993] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5: : Instructing follower be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:25.880712 30406 raft_consensus.cc:1081] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Signalling peer be90690510254d95a22a7def29abc3c4 to start an election
I20251025 20:03:25.882121 30096 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
from {username='slave'} at 127.0.0.1:55422
I20251025 20:03:25.882586 30096 raft_consensus.cc:493] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:25.882841 30096 raft_consensus.cc:3060] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:25.887630 30096 raft_consensus.cc:515] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.889063 30096 leader_election.cc:290] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 2 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:25.890141 30171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:25.890141 30246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528" candidate_uuid: "be90690510254d95a22a7def29abc3c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:25.890895 30246 raft_consensus.cc:3060] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:25.890895 30171 raft_consensus.cc:3055] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:25.891426 30171 raft_consensus.cc:740] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:25.892285 30171 consensus_queue.cc:260] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.893468 30171 raft_consensus.cc:3060] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:25.898269 30246 raft_consensus.cc:2468] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 2.
I20251025 20:03:25.899392 30060 leader_election.cc:304] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be90690510254d95a22a7def29abc3c4, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:25.900104 30412 raft_consensus.cc:2804] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:25.900213 30171 raft_consensus.cc:2468] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be90690510254d95a22a7def29abc3c4 in term 2.
I20251025 20:03:25.900619 30412 raft_consensus.cc:697] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Becoming Leader. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:25.901512 30412 consensus_queue.cc:237] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:25.908818 29995 catalog_manager.cc:5649] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 reported cstate change: term changed from 1 to 2, leader changed from 17d90a5990d3424db25cf46b32327df5 (127.28.192.194) to be90690510254d95a22a7def29abc3c4 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "be90690510254d95a22a7def29abc3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:25.971585 30246 raft_consensus.cc:1275] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:03:25.972636 30412 consensus_queue.cc:1048] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:03:25.983105 30171 raft_consensus.cc:1275] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:03:25.984656 30419 consensus_queue.cc:1048] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:03:26.047658 30246 raft_consensus.cc:1275] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:26.048786 30419 consensus_queue.cc:1048] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:26.058080 30171 raft_consensus.cc:1275] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:26.059962 30419 consensus_queue.cc:1048] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:26.179435 30171 raft_consensus.cc:1275] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:03:26.180756 30419 consensus_queue.cc:1048] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:03:26.187351 30246 raft_consensus.cc:1275] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:03:26.189162 30419 consensus_queue.cc:1048] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:03:26.192267 30246 raft_consensus.cc:1275] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:03:26.193840 30412 consensus_queue.cc:1048] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:03:26.203496 30171 raft_consensus.cc:1275] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:03:26.206352 30412 consensus_queue.cc:1048] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:03:26.384974 30171 raft_consensus.cc:1275] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:26.386291 30412 consensus_queue.cc:1048] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:26.394713 30244 raft_consensus.cc:1275] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Refusing update from remote peer be90690510254d95a22a7def29abc3c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:26.396546 30419 consensus_queue.cc:1048] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:28.532269 29443 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20251025 20:03:28.533134 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:28.540863 30096 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
mode: GRACEFUL
new_leader_uuid: "d00e76da60c94a3482243bba69caeee7"
from {username='slave'} at 127.0.0.1:55446
I20251025 20:03:28.541669 30096 raft_consensus.cc:606] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Received request to transfer leadership to TS d00e76da60c94a3482243bba69caeee7
I20251025 20:03:28.544608 30096 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
mode: GRACEFUL
new_leader_uuid: "d00e76da60c94a3482243bba69caeee7"
from {username='slave'} at 127.0.0.1:55446
I20251025 20:03:28.545053 30096 raft_consensus.cc:606] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Received request to transfer leadership to TS d00e76da60c94a3482243bba69caeee7
I20251025 20:03:28.547130 30096 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
mode: GRACEFUL
new_leader_uuid: "17d90a5990d3424db25cf46b32327df5"
from {username='slave'} at 127.0.0.1:55446
I20251025 20:03:28.547575 30096 raft_consensus.cc:606] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Received request to transfer leadership to TS 17d90a5990d3424db25cf46b32327df5
I20251025 20:03:28.549713 30096 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528"
dest_uuid: "be90690510254d95a22a7def29abc3c4"
mode: GRACEFUL
new_leader_uuid: "17d90a5990d3424db25cf46b32327df5"
from {username='slave'} at 127.0.0.1:55446
I20251025 20:03:28.550271 30096 raft_consensus.cc:606] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Received request to transfer leadership to TS 17d90a5990d3424db25cf46b32327df5
I20251025 20:03:28.551213 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251025 20:03:28.551802 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:28.610163 30440 raft_consensus.cc:993] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4: : Instructing follower 17d90a5990d3424db25cf46b32327df5 to start an election
I20251025 20:03:28.610809 30419 raft_consensus.cc:1081] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Signalling peer 17d90a5990d3424db25cf46b32327df5 to start an election
I20251025 20:03:28.612325 30172 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528"
dest_uuid: "17d90a5990d3424db25cf46b32327df5"
from {username='slave'} at 127.0.0.1:41428
I20251025 20:03:28.612828 30172 raft_consensus.cc:493] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:28.613118 30172 raft_consensus.cc:3060] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:28.620671 30172 raft_consensus.cc:515] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:28.623036 30172 leader_election.cc:290] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 3 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:28.623901 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:28.624234 30244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df3103d1e0b44cd79fc56a331af83528" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:28.624672 30096 raft_consensus.cc:3055] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:03:28.624784 30244 raft_consensus.cc:3060] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:28.624995 30096 raft_consensus.cc:740] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:28.625860 30096 consensus_queue.cc:260] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:28.627018 30096 raft_consensus.cc:3060] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:28.630556 30244 raft_consensus.cc:2468] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 3.
I20251025 20:03:28.631754 30135 leader_election.cc:304] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:28.632594 30096 raft_consensus.cc:2468] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 3.
I20251025 20:03:28.632787 30406 raft_consensus.cc:2804] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:28.633857 30406 raft_consensus.cc:697] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 3 LEADER]: Becoming Leader. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:28.634727 30406 consensus_queue.cc:237] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:28.643009 29993 catalog_manager.cc:5649] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 reported cstate change: term changed from 2 to 3, leader changed from be90690510254d95a22a7def29abc3c4 (127.28.192.193) to 17d90a5990d3424db25cf46b32327df5 (127.28.192.194). New cstate: current_term: 3 leader_uuid: "17d90a5990d3424db25cf46b32327df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:28.732985 30440 raft_consensus.cc:993] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4: : Instructing follower d00e76da60c94a3482243bba69caeee7 to start an election
I20251025 20:03:28.733435 30419 raft_consensus.cc:1081] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Signalling peer d00e76da60c94a3482243bba69caeee7 to start an election
I20251025 20:03:28.734875 30244 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7"
dest_uuid: "d00e76da60c94a3482243bba69caeee7"
from {username='slave'} at 127.0.0.1:58198
I20251025 20:03:28.735347 30244 raft_consensus.cc:493] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:28.735626 30244 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:28.740406 30244 raft_consensus.cc:515] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:28.742383 30244 leader_election.cc:290] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 4 election: Requested vote from peers 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617), be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415)
I20251025 20:03:28.743535 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:28.743515 30170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "728e3db5ac604f77b56f8e45c9520bf7" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:28.744215 30096 raft_consensus.cc:3055] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:03:28.744364 30170 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:28.744585 30096 raft_consensus.cc:740] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:28.745458 30096 consensus_queue.cc:260] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:28.746914 30096 raft_consensus.cc:3060] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:28.753352 30170 raft_consensus.cc:2468] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 4.
I20251025 20:03:28.754619 30096 raft_consensus.cc:2468] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 4.
I20251025 20:03:28.755199 30211 leader_election.cc:304] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:28.756286 30411 raft_consensus.cc:2804] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:03:28.756426 30440 raft_consensus.cc:993] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4: : Instructing follower d00e76da60c94a3482243bba69caeee7 to start an election
I20251025 20:03:28.756892 30419 raft_consensus.cc:1081] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Signalling peer d00e76da60c94a3482243bba69caeee7 to start an election
I20251025 20:03:28.757761 30411 raft_consensus.cc:697] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 4 LEADER]: Becoming Leader. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:28.758596 30244 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b"
dest_uuid: "d00e76da60c94a3482243bba69caeee7"
from {username='slave'} at 127.0.0.1:58198
I20251025 20:03:28.759037 30244 raft_consensus.cc:493] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:28.759303 30244 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:28.758903 30411 consensus_queue.cc:237] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } }
I20251025 20:03:28.765293 30244 raft_consensus.cc:515] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:28.767165 30244 leader_election.cc:290] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 4 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617)
I20251025 20:03:28.767599 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:28.767786 30170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45e270333ba34fd78b5b7c96a303e33b" candidate_uuid: "d00e76da60c94a3482243bba69caeee7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "17d90a5990d3424db25cf46b32327df5"
I20251025 20:03:28.768170 30096 raft_consensus.cc:3055] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:03:28.767429 29993 catalog_manager.cc:5649] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 reported cstate change: term changed from 3 to 4, leader changed from be90690510254d95a22a7def29abc3c4 (127.28.192.193) to d00e76da60c94a3482243bba69caeee7 (127.28.192.195). New cstate: current_term: 4 leader_uuid: "d00e76da60c94a3482243bba69caeee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:28.768316 30170 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:28.768519 30096 raft_consensus.cc:740] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:28.769318 30096 consensus_queue.cc:260] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:28.770399 30096 raft_consensus.cc:3060] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:28.775799 30170 raft_consensus.cc:2468] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 4.
I20251025 20:03:28.776422 30096 raft_consensus.cc:2468] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e76da60c94a3482243bba69caeee7 in term 4.
I20251025 20:03:28.777210 30211 leader_election.cc:304] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:28.778152 30411 raft_consensus.cc:2804] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:03:28.778646 30411 raft_consensus.cc:697] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 4 LEADER]: Becoming Leader. State: Replica: d00e76da60c94a3482243bba69caeee7, State: Running, Role: LEADER
I20251025 20:03:28.779371 30411 consensus_queue.cc:237] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:28.786994 29993 catalog_manager.cc:5649] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 reported cstate change: term changed from 3 to 4, leader changed from be90690510254d95a22a7def29abc3c4 (127.28.192.193) to d00e76da60c94a3482243bba69caeee7 (127.28.192.195). New cstate: current_term: 4 leader_uuid: "d00e76da60c94a3482243bba69caeee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:28.873328 30440 raft_consensus.cc:993] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4: : Instructing follower 17d90a5990d3424db25cf46b32327df5 to start an election
I20251025 20:03:28.873800 30419 raft_consensus.cc:1081] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Signalling peer 17d90a5990d3424db25cf46b32327df5 to start an election
I20251025 20:03:28.875969 30170 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8"
dest_uuid: "17d90a5990d3424db25cf46b32327df5"
from {username='slave'} at 127.0.0.1:41428
I20251025 20:03:28.876571 30170 raft_consensus.cc:493] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251025 20:03:28.876891 30170 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:28.883090 30170 raft_consensus.cc:515] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:28.885071 30170 leader_election.cc:290] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 4 election: Requested vote from peers be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415), d00e76da60c94a3482243bba69caeee7 (127.28.192.195:36453)
I20251025 20:03:28.886040 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "be90690510254d95a22a7def29abc3c4"
I20251025 20:03:28.886196 30244 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceb28906ae414e8dac5ceea5f98ae1a8" candidate_uuid: "17d90a5990d3424db25cf46b32327df5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d00e76da60c94a3482243bba69caeee7"
I20251025 20:03:28.886662 30096 raft_consensus.cc:3055] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:03:28.886740 30244 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:28.886965 30096 raft_consensus.cc:740] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be90690510254d95a22a7def29abc3c4, State: Running, Role: LEADER
I20251025 20:03:28.887669 30096 consensus_queue.cc:260] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:28.888759 30096 raft_consensus.cc:3060] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:28.892271 30244 raft_consensus.cc:2468] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 4.
I20251025 20:03:28.893327 30135 leader_election.cc:304] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d90a5990d3424db25cf46b32327df5, d00e76da60c94a3482243bba69caeee7; no voters:
I20251025 20:03:28.893792 30096 raft_consensus.cc:2468] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d90a5990d3424db25cf46b32327df5 in term 4.
I20251025 20:03:28.894110 30406 raft_consensus.cc:2804] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:03:28.894622 30406 raft_consensus.cc:697] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 4 LEADER]: Becoming Leader. State: Replica: 17d90a5990d3424db25cf46b32327df5, State: Running, Role: LEADER
I20251025 20:03:28.895534 30406 consensus_queue.cc:237] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } }
I20251025 20:03:28.904067 29993 catalog_manager.cc:5649] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 reported cstate change: term changed from 3 to 4, leader changed from be90690510254d95a22a7def29abc3c4 (127.28.192.193) to 17d90a5990d3424db25cf46b32327df5 (127.28.192.194). New cstate: current_term: 4 leader_uuid: "17d90a5990d3424db25cf46b32327df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:29.187692 30244 raft_consensus.cc:1275] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Refusing update from remote peer 17d90a5990d3424db25cf46b32327df5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:03:29.189103 30406 consensus_queue.cc:1048] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:03:29.200284 30096 raft_consensus.cc:1275] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Refusing update from remote peer 17d90a5990d3424db25cf46b32327df5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:03:29.201937 30406 consensus_queue.cc:1048] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:03:29.203966 30170 raft_consensus.cc:1275] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Refusing update from remote peer d00e76da60c94a3482243bba69caeee7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:03:29.205358 30411 consensus_queue.cc:1048] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:03:29.217756 30096 raft_consensus.cc:1275] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Refusing update from remote peer d00e76da60c94a3482243bba69caeee7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:03:29.220837 30411 consensus_queue.cc:1048] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:03:29.396622 30170 raft_consensus.cc:1275] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Refusing update from remote peer d00e76da60c94a3482243bba69caeee7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:03:29.396622 30096 raft_consensus.cc:1275] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Refusing update from remote peer d00e76da60c94a3482243bba69caeee7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:03:29.398164 30447 consensus_queue.cc:1048] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d90a5990d3424db25cf46b32327df5" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:03:29.398963 30457 consensus_queue.cc:1048] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:03:29.431959 30244 raft_consensus.cc:1275] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 4 FOLLOWER]: Refusing update from remote peer 17d90a5990d3424db25cf46b32327df5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:03:29.433228 30441 consensus_queue.cc:1048] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e76da60c94a3482243bba69caeee7" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:03:29.442407 30096 raft_consensus.cc:1275] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Refusing update from remote peer 17d90a5990d3424db25cf46b32327df5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:03:29.444362 30462 consensus_queue.cc:1048] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be90690510254d95a22a7def29abc3c4" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251025 20:03:29.552857 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:29.558370 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:29.558820 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:30.559793 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:30.565232 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:30.565689 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:31.566686 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:31.572093 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:31.572628 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:32.573704 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:32.579344 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:32.579860 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:33.580963 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:33.587080 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:33.587651 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:34.588770 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:34.594493 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:34.595034 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:35.596215 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:35.602993 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:35.603441 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:36.604393 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:36.610168 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:36.610631 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:37.611979 29443 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251025 20:03:37.618793 29443 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251025 20:03:37.619262 29443 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251025 20:03:38.648902 29443 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20251025 20:03:38.649411 29443 tablet_server.cc:178] TabletServer@127.28.192.193:0 shutting down...
I20251025 20:03:38.680975 29443 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251025 20:03:38.681870 29443 tablet_replica.cc:333] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4: stopping tablet replica
I20251025 20:03:38.682679 29443 raft_consensus.cc:2243] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Raft consensus shutting down.
I20251025 20:03:38.683758 29443 raft_consensus.cc:2272] T 2038c9436df6436d92f7a25bfc5474bf P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.686496 29443 tablet_replica.cc:333] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4: stopping tablet replica
I20251025 20:03:38.687186 29443 raft_consensus.cc:2243] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Raft consensus shutting down.
I20251025 20:03:38.688319 29443 raft_consensus.cc:2272] T ea6745d0705144b0999e3a2f7a35e8da P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.690853 29443 tablet_replica.cc:333] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4: stopping tablet replica
I20251025 20:03:38.691339 29443 raft_consensus.cc:2243] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.691766 29443 raft_consensus.cc:2272] T df3103d1e0b44cd79fc56a331af83528 P be90690510254d95a22a7def29abc3c4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.693543 29443 tablet_replica.cc:333] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4: stopping tablet replica
I20251025 20:03:38.694043 29443 raft_consensus.cc:2243] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Raft consensus shutting down.
I20251025 20:03:38.694826 29443 raft_consensus.cc:2272] T 38e64303be304c388287465e021ab853 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.696653 29443 tablet_replica.cc:333] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4: stopping tablet replica
I20251025 20:03:38.697229 29443 raft_consensus.cc:2243] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 2 LEADER]: Raft consensus shutting down.
I20251025 20:03:38.698096 29443 raft_consensus.cc:2272] T 5961f6062df94219941afc0503ff3033 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.699858 29443 tablet_replica.cc:333] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4: stopping tablet replica
I20251025 20:03:38.700340 29443 raft_consensus.cc:2243] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.700806 29443 raft_consensus.cc:2272] T 728e3db5ac604f77b56f8e45c9520bf7 P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.702826 29443 tablet_replica.cc:333] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4: stopping tablet replica
I20251025 20:03:38.703378 29443 raft_consensus.cc:2243] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.703802 29443 raft_consensus.cc:2272] T 45e270333ba34fd78b5b7c96a303e33b P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.705636 29443 tablet_replica.cc:333] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4: stopping tablet replica
I20251025 20:03:38.706211 29443 raft_consensus.cc:2243] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.706632 29443 raft_consensus.cc:2272] T 4128ceefc2c541ccb009b58235c071e6 P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.708540 29443 tablet_replica.cc:333] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4: stopping tablet replica
I20251025 20:03:38.709007 29443 raft_consensus.cc:2243] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.709407 29443 raft_consensus.cc:2272] T ceb28906ae414e8dac5ceea5f98ae1a8 P be90690510254d95a22a7def29abc3c4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.711232 29443 tablet_replica.cc:333] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4: stopping tablet replica
I20251025 20:03:38.711679 29443 raft_consensus.cc:2243] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.712133 29443 raft_consensus.cc:2272] T 7ce5274ff7fe43fab967b217512f197d P be90690510254d95a22a7def29abc3c4 [term 2 FOLLOWER]: Raft consensus is shut down!
W20251025 20:03:38.744071 30135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.192.193:34415: connect: Connection refused (error 111) [suppressed 4 similar messages]
I20251025 20:03:38.745973 29443 tablet_server.cc:195] TabletServer@127.28.192.193:0 shutdown complete.
W20251025 20:03:38.748131 30135 consensus_peers.cc:597] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 -> Peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Couldn't send request to peer be90690510254d95a22a7def29abc3c4. Status: Network error: Client connection negotiation failed: client connection to 127.28.192.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:38.771456 29443 tablet_server.cc:178] TabletServer@127.28.192.194:0 shutting down...
W20251025 20:03:38.792378 30209 consensus_peers.cc:597] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 -> Peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Couldn't send request to peer be90690510254d95a22a7def29abc3c4. Status: Network error: Client connection negotiation failed: client connection to 127.28.192.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:38.804793 29443 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251025 20:03:38.805609 29443 tablet_replica.cc:333] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5: stopping tablet replica
I20251025 20:03:38.806463 29443 raft_consensus.cc:2243] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.807000 29443 raft_consensus.cc:2272] T 7ce5274ff7fe43fab967b217512f197d P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.809033 29443 tablet_replica.cc:333] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5: stopping tablet replica
I20251025 20:03:38.809621 29443 raft_consensus.cc:2243] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 2 LEADER]: Raft consensus shutting down.
I20251025 20:03:38.810587 29443 raft_consensus.cc:2272] T 4128ceefc2c541ccb009b58235c071e6 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.812448 29443 tablet_replica.cc:333] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5: stopping tablet replica
I20251025 20:03:38.812969 29443 raft_consensus.cc:2243] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 4 LEADER]: Raft consensus shutting down.
I20251025 20:03:38.813817 29443 raft_consensus.cc:2272] T ceb28906ae414e8dac5ceea5f98ae1a8 P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.815783 29443 tablet_replica.cc:333] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5: stopping tablet replica
I20251025 20:03:38.816298 29443 raft_consensus.cc:2243] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.816732 29443 raft_consensus.cc:2272] T 45e270333ba34fd78b5b7c96a303e33b P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.818493 29443 tablet_replica.cc:333] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5: stopping tablet replica
I20251025 20:03:38.819005 29443 raft_consensus.cc:2243] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 3 LEADER]: Raft consensus shutting down.
I20251025 20:03:38.819833 29443 raft_consensus.cc:2272] T df3103d1e0b44cd79fc56a331af83528 P 17d90a5990d3424db25cf46b32327df5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.821561 29443 tablet_replica.cc:333] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5: stopping tablet replica
I20251025 20:03:38.822155 29443 raft_consensus.cc:2243] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.822578 29443 raft_consensus.cc:2272] T ea6745d0705144b0999e3a2f7a35e8da P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.824481 29443 tablet_replica.cc:333] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5: stopping tablet replica
I20251025 20:03:38.825006 29443 raft_consensus.cc:2243] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.825420 29443 raft_consensus.cc:2272] T 2038c9436df6436d92f7a25bfc5474bf P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.827215 29443 tablet_replica.cc:333] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5: stopping tablet replica
I20251025 20:03:38.827723 29443 raft_consensus.cc:2243] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.828137 29443 raft_consensus.cc:2272] T 5961f6062df94219941afc0503ff3033 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.829973 29443 tablet_replica.cc:333] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5: stopping tablet replica
I20251025 20:03:38.830425 29443 raft_consensus.cc:2243] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.830844 29443 raft_consensus.cc:2272] T 728e3db5ac604f77b56f8e45c9520bf7 P 17d90a5990d3424db25cf46b32327df5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.833304 29443 tablet_replica.cc:333] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5: stopping tablet replica
I20251025 20:03:38.833776 29443 raft_consensus.cc:2243] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.834405 29443 raft_consensus.cc:2272] T 38e64303be304c388287465e021ab853 P 17d90a5990d3424db25cf46b32327df5 [term 2 FOLLOWER]: Raft consensus is shut down!
W20251025 20:03:38.836081 30209 consensus_peers.cc:597] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 -> Peer be90690510254d95a22a7def29abc3c4 (127.28.192.193:34415): Couldn't send request to peer be90690510254d95a22a7def29abc3c4. Status: Network error: Client connection negotiation failed: client connection to 127.28.192.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251025 20:03:38.849498 30135 connection.cc:441] server connection from 127.0.0.1:41414 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41414 (request call id 156) could send its response
W20251025 20:03:38.853245 30211 consensus_peers.cc:597] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 -> Peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Couldn't send request to peer 17d90a5990d3424db25cf46b32327df5. Status: Network error: Client connection negotiation failed: client connection to 127.28.192.194:44617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:38.866528 29443 tablet_server.cc:195] TabletServer@127.28.192.194:0 shutdown complete.
W20251025 20:03:38.877079 30211 consensus_peers.cc:597] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 -> Peer 17d90a5990d3424db25cf46b32327df5 (127.28.192.194:44617): Couldn't send request to peer 17d90a5990d3424db25cf46b32327df5. Status: Network error: Client connection negotiation failed: client connection to 127.28.192.194:44617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:38.893333 29443 tablet_server.cc:178] TabletServer@127.28.192.195:0 shutting down...
I20251025 20:03:38.922463 29443 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251025 20:03:38.923206 29443 tablet_replica.cc:333] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7: stopping tablet replica
I20251025 20:03:38.923959 29443 raft_consensus.cc:2243] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.924579 29443 raft_consensus.cc:2272] T 5961f6062df94219941afc0503ff3033 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.926816 29443 tablet_replica.cc:333] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7: stopping tablet replica
I20251025 20:03:38.927382 29443 raft_consensus.cc:2243] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 4 LEADER]: Raft consensus shutting down.
I20251025 20:03:38.928282 29443 raft_consensus.cc:2272] T 728e3db5ac604f77b56f8e45c9520bf7 P d00e76da60c94a3482243bba69caeee7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.930236 29443 tablet_replica.cc:333] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7: stopping tablet replica
I20251025 20:03:38.930753 29443 raft_consensus.cc:2243] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 2 LEADER]: Raft consensus shutting down.
I20251025 20:03:38.931558 29443 raft_consensus.cc:2272] T 7ce5274ff7fe43fab967b217512f197d P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.933391 29443 tablet_replica.cc:333] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7: stopping tablet replica
I20251025 20:03:38.933944 29443 raft_consensus.cc:2243] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 4 LEADER]: Raft consensus shutting down.
I20251025 20:03:38.934727 29443 raft_consensus.cc:2272] T 45e270333ba34fd78b5b7c96a303e33b P d00e76da60c94a3482243bba69caeee7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.936584 29443 tablet_replica.cc:333] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7: stopping tablet replica
I20251025 20:03:38.937134 29443 raft_consensus.cc:2243] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.937533 29443 raft_consensus.cc:2272] T 38e64303be304c388287465e021ab853 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.939260 29443 tablet_replica.cc:333] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7: stopping tablet replica
I20251025 20:03:38.939790 29443 raft_consensus.cc:2243] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.940258 29443 raft_consensus.cc:2272] T ceb28906ae414e8dac5ceea5f98ae1a8 P d00e76da60c94a3482243bba69caeee7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.942034 29443 tablet_replica.cc:333] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7: stopping tablet replica
I20251025 20:03:38.942767 29443 raft_consensus.cc:2243] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.943219 29443 raft_consensus.cc:2272] T 4128ceefc2c541ccb009b58235c071e6 P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.945079 29443 tablet_replica.cc:333] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7: stopping tablet replica
I20251025 20:03:38.945612 29443 raft_consensus.cc:2243] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.946116 29443 raft_consensus.cc:2272] T ea6745d0705144b0999e3a2f7a35e8da P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.948096 29443 tablet_replica.cc:333] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7: stopping tablet replica
I20251025 20:03:38.948582 29443 raft_consensus.cc:2243] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.949036 29443 raft_consensus.cc:2272] T df3103d1e0b44cd79fc56a331af83528 P d00e76da60c94a3482243bba69caeee7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.950840 29443 tablet_replica.cc:333] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7: stopping tablet replica
I20251025 20:03:38.951303 29443 raft_consensus.cc:2243] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:03:38.951712 29443 raft_consensus.cc:2272] T 2038c9436df6436d92f7a25bfc5474bf P d00e76da60c94a3482243bba69caeee7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:38.982051 29443 tablet_server.cc:195] TabletServer@127.28.192.195:0 shutdown complete.
I20251025 20:03:39.004818 29443 master.cc:561] Master@127.28.192.254:44611 shutting down...
I20251025 20:03:39.029675 29443 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:03:39.030403 29443 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:03:39.030807 29443 tablet_replica.cc:333] T 00000000000000000000000000000000 P ef8be305b62d409cb81816f1f11d62f8: stopping tablet replica
I20251025 20:03:39.053124 29443 master.cc:583] Master@127.28.192.254:44611 shutdown complete.
17d90a5990d3424db25cf46b32327df5 4
be90690510254d95a22a7def29abc3c4 5
d00e76da60c94a3482243bba69caeee7 1
17d90a5990d3424db25cf46b32327df5 4
be90690510254d95a22a7def29abc3c4 3
d00e76da60c94a3482243bba69caeee7 3
17d90a5990d3424db25cf46b32327df5 1
be90690510254d95a22a7def29abc3c4 8
d00e76da60c94a3482243bba69caeee7 1
17d90a5990d3424db25cf46b32327df5 3
be90690510254d95a22a7def29abc3c4 4
d00e76da60c94a3482243bba69caeee7 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28792 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20251025 20:03:39.100144 29443 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.192.254:41145
I20251025 20:03:39.101436 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:03:39.107465 30477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:39.107582 30478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:39.108531 30480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:03:39.109879 29443 server_base.cc:1047] running on GCE node
I20251025 20:03:39.110980 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:03:39.111191 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:03:39.111336 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422619111316 us; error 0 us; skew 500 ppm
I20251025 20:03:39.111893 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:03:39.114764 29443 webserver.cc:492] Webserver started at http://127.28.192.254:32849/ using document root <none> and password file <none>
I20251025 20:03:39.115312 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:03:39.115523 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:03:39.115818 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:03:39.117208 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/master-0-root/instance:
uuid: "5b68518dd72d4e87a5c2dcf7c969fe49"
format_stamp: "Formatted at 2025-10-25 20:03:39 on dist-test-slave-svnn"
I20251025 20:03:39.122450 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20251025 20:03:39.126351 30485 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:39.127159 29443 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20251025 20:03:39.127449 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/master-0-root
uuid: "5b68518dd72d4e87a5c2dcf7c969fe49"
format_stamp: "Formatted at 2025-10-25 20:03:39 on dist-test-slave-svnn"
I20251025 20:03:39.127755 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:03:39.146924 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:03:39.148335 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:03:39.192499 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.254:41145
I20251025 20:03:39.192596 30536 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.254:41145 every 8 connection(s)
I20251025 20:03:39.196812 30537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.208942 30537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Bootstrap starting.
I20251025 20:03:39.214234 30537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:39.219098 30537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: No bootstrap required, opened a new log
I20251025 20:03:39.221504 30537 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b68518dd72d4e87a5c2dcf7c969fe49" member_type: VOTER }
I20251025 20:03:39.222018 30537 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:39.222270 30537 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b68518dd72d4e87a5c2dcf7c969fe49, State: Initialized, Role: FOLLOWER
I20251025 20:03:39.222856 30537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b68518dd72d4e87a5c2dcf7c969fe49" member_type: VOTER }
I20251025 20:03:39.223323 30537 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:03:39.223544 30537 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:03:39.223832 30537 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:39.228750 30537 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b68518dd72d4e87a5c2dcf7c969fe49" member_type: VOTER }
I20251025 20:03:39.229375 30537 leader_election.cc:304] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b68518dd72d4e87a5c2dcf7c969fe49; no voters:
I20251025 20:03:39.230710 30537 leader_election.cc:290] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:03:39.231140 30540 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:39.232529 30540 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [term 1 LEADER]: Becoming Leader. State: Replica: 5b68518dd72d4e87a5c2dcf7c969fe49, State: Running, Role: LEADER
I20251025 20:03:39.233255 30540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b68518dd72d4e87a5c2dcf7c969fe49" member_type: VOTER }
I20251025 20:03:39.234108 30537 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:03:39.239391 30542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b68518dd72d4e87a5c2dcf7c969fe49. Latest consensus state: current_term: 1 leader_uuid: "5b68518dd72d4e87a5c2dcf7c969fe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b68518dd72d4e87a5c2dcf7c969fe49" member_type: VOTER } }
I20251025 20:03:39.240254 30542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [sys.catalog]: This master's current role is: LEADER
I20251025 20:03:39.241433 30541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b68518dd72d4e87a5c2dcf7c969fe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b68518dd72d4e87a5c2dcf7c969fe49" member_type: VOTER } }
I20251025 20:03:39.242755 30541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49 [sys.catalog]: This master's current role is: LEADER
I20251025 20:03:39.249279 30548 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:03:39.254922 30548 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:03:39.256119 29443 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251025 20:03:39.265388 30548 catalog_manager.cc:1357] Generated new cluster ID: ddaa2ab3c03949edacb8f7c78ac3eb42
I20251025 20:03:39.265759 30548 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:03:39.282773 30548 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:03:39.284224 30548 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:03:39.304334 30548 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Generated new TSK 0
I20251025 20:03:39.305114 30548 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 20:03:39.323117 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:03:39.329711 30558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:39.331135 30559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:39.334431 30561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:03:39.334980 29443 server_base.cc:1047] running on GCE node
I20251025 20:03:39.336061 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:03:39.336256 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:03:39.336381 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422619336370 us; error 0 us; skew 500 ppm
I20251025 20:03:39.336910 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:03:39.339545 29443 webserver.cc:492] Webserver started at http://127.28.192.193:39823/ using document root <none> and password file <none>
I20251025 20:03:39.340068 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:03:39.340240 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:03:39.340463 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:03:39.341674 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-0-root/instance:
uuid: "349d66b543bf47249ac78154b908b957"
format_stamp: "Formatted at 2025-10-25 20:03:39 on dist-test-slave-svnn"
I20251025 20:03:39.346844 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251025 20:03:39.350497 30566 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:39.351392 29443 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 20:03:39.351696 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-0-root
uuid: "349d66b543bf47249ac78154b908b957"
format_stamp: "Formatted at 2025-10-25 20:03:39 on dist-test-slave-svnn"
I20251025 20:03:39.352008 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:03:39.363392 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:03:39.364802 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:03:39.366451 29443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:03:39.368932 29443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:03:39.369138 29443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:39.369375 29443 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:03:39.369551 29443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:39.416347 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.193:33717
I20251025 20:03:39.416445 30628 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.193:33717 every 8 connection(s)
I20251025 20:03:39.421334 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:03:39.430183 30633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:39.433010 30634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:39.434614 30636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:03:39.434758 29443 server_base.cc:1047] running on GCE node
I20251025 20:03:39.436502 30629 heartbeater.cc:344] Connected to a master server at 127.28.192.254:41145
I20251025 20:03:39.436883 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20251025 20:03:39.436905 30629 heartbeater.cc:461] Registering TS with master...
W20251025 20:03:39.437168 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:03:39.437342 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422619437325 us; error 0 us; skew 500 ppm
I20251025 20:03:39.437824 30629 heartbeater.cc:507] Master 127.28.192.254:41145 requested a full tablet report, sending...
I20251025 20:03:39.438036 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:03:39.440233 30502 ts_manager.cc:194] Registered new tserver with Master: 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717)
I20251025 20:03:39.441144 29443 webserver.cc:492] Webserver started at http://127.28.192.194:40335/ using document root <none> and password file <none>
I20251025 20:03:39.441699 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:03:39.441932 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:03:39.442204 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:03:39.442409 30502 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39160
I20251025 20:03:39.443575 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-1-root/instance:
uuid: "583af532dff04e8ebba45013323d516d"
format_stamp: "Formatted at 2025-10-25 20:03:39 on dist-test-slave-svnn"
I20251025 20:03:39.449067 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251025 20:03:39.452747 30641 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:39.453572 29443 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:03:39.453933 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-1-root
uuid: "583af532dff04e8ebba45013323d516d"
format_stamp: "Formatted at 2025-10-25 20:03:39 on dist-test-slave-svnn"
I20251025 20:03:39.454226 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:03:39.469100 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:03:39.470511 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:03:39.472075 29443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:03:39.474548 29443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:03:39.474741 29443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:39.474992 29443 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:03:39.475204 29443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:39.520042 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.194:32975
I20251025 20:03:39.520150 30703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.194:32975 every 8 connection(s)
I20251025 20:03:39.525278 29443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:03:39.532219 30707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:03:39.534790 30708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:03:39.536016 29443 server_base.cc:1047] running on GCE node
W20251025 20:03:39.537130 30710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:03:39.538342 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251025 20:03:39.538573 29443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251025 20:03:39.538758 29443 hybrid_clock.cc:648] HybridClock initialized: now 1761422619538736 us; error 0 us; skew 500 ppm
I20251025 20:03:39.538926 30704 heartbeater.cc:344] Connected to a master server at 127.28.192.254:41145
I20251025 20:03:39.539288 30704 heartbeater.cc:461] Registering TS with master...
I20251025 20:03:39.539399 29443 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 20:03:39.540086 30704 heartbeater.cc:507] Master 127.28.192.254:41145 requested a full tablet report, sending...
I20251025 20:03:39.542313 29443 webserver.cc:492] Webserver started at http://127.28.192.195:41923/ using document root <none> and password file <none>
I20251025 20:03:39.542433 30502 ts_manager.cc:194] Registered new tserver with Master: 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:39.542866 29443 fs_manager.cc:362] Metadata directory not provided
I20251025 20:03:39.543088 29443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:03:39.543335 29443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:03:39.543977 30502 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39172
I20251025 20:03:39.544660 29443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-2-root/instance:
uuid: "783f8fd6d80141b7912f66a954a6f92e"
format_stamp: "Formatted at 2025-10-25 20:03:39 on dist-test-slave-svnn"
I20251025 20:03:39.549670 29443 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20251025 20:03:39.553289 30715 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:39.554214 29443 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 20:03:39.554482 29443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-2-root
uuid: "783f8fd6d80141b7912f66a954a6f92e"
format_stamp: "Formatted at 2025-10-25 20:03:39 on dist-test-slave-svnn"
I20251025 20:03:39.554744 29443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:03:39.568099 29443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:03:39.569417 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:03:39.571019 29443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:03:39.573408 29443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:03:39.573597 29443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:03:39.573868 29443 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:03:39.574064 29443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251025 20:03:39.619091 29443 rpc_server.cc:307] RPC server started. Bound to: 127.28.192.195:46541
I20251025 20:03:39.619196 30777 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.192.195:46541 every 8 connection(s)
I20251025 20:03:39.634351 30778 heartbeater.cc:344] Connected to a master server at 127.28.192.254:41145
I20251025 20:03:39.634884 30778 heartbeater.cc:461] Registering TS with master...
I20251025 20:03:39.635715 30778 heartbeater.cc:507] Master 127.28.192.254:41145 requested a full tablet report, sending...
I20251025 20:03:39.637777 30502 ts_manager.cc:194] Registered new tserver with Master: 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:39.638119 29443 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015584831s
I20251025 20:03:39.639302 30502 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39186
I20251025 20:03:39.640384 29443 test_util.cc:276] Using random seed: 696975911
I20251025 20:03:39.666245 30502 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39188:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:03:39.672942 30502 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:03:39.897451 30743 tablet_service.cc:1505] Processing CreateTablet for tablet 78afc6e697b1482391aa5245bee3d881 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251025 20:03:39.899421 30743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78afc6e697b1482391aa5245bee3d881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.902657 30742 tablet_service.cc:1505] Processing CreateTablet for tablet 13be59a31cda407db6fa50a400f37e1f (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251025 20:03:39.904458 30742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13be59a31cda407db6fa50a400f37e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.907168 30669 tablet_service.cc:1505] Processing CreateTablet for tablet 78afc6e697b1482391aa5245bee3d881 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251025 20:03:39.908720 30669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78afc6e697b1482391aa5245bee3d881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.915802 30668 tablet_service.cc:1505] Processing CreateTablet for tablet 13be59a31cda407db6fa50a400f37e1f (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251025 20:03:39.917414 30668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13be59a31cda407db6fa50a400f37e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.919103 30741 tablet_service.cc:1505] Processing CreateTablet for tablet da4360132dff42a18667801272f14797 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251025 20:03:39.921955 30741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4360132dff42a18667801272f14797. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.923885 30667 tablet_service.cc:1505] Processing CreateTablet for tablet da4360132dff42a18667801272f14797 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251025 20:03:39.925480 30667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4360132dff42a18667801272f14797. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.926386 30793 tablet_bootstrap.cc:492] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:39.929515 30666 tablet_service.cc:1505] Processing CreateTablet for tablet 06d7fa7a698c4890af235aed97fd6fb5 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251025 20:03:39.931113 30666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d7fa7a698c4890af235aed97fd6fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.936271 30743 tablet_service.cc:1505] Processing CreateTablet for tablet 06d7fa7a698c4890af235aed97fd6fb5 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251025 20:03:39.936808 30740 tablet_service.cc:1505] Processing CreateTablet for tablet b406b18ee0364eb5b4e35826fe7290be (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251025 20:03:39.938041 30743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d7fa7a698c4890af235aed97fd6fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.940217 30739 tablet_service.cc:1505] Processing CreateTablet for tablet 3a03f8a32da8499dbc0e2a426d214001 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251025 20:03:39.940729 30665 tablet_service.cc:1505] Processing CreateTablet for tablet b406b18ee0364eb5b4e35826fe7290be (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251025 20:03:39.941823 30739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a03f8a32da8499dbc0e2a426d214001. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.942171 30665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b406b18ee0364eb5b4e35826fe7290be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.945942 30740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b406b18ee0364eb5b4e35826fe7290be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.946043 30793 tablet_bootstrap.cc:654] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:39.955736 30664 tablet_service.cc:1505] Processing CreateTablet for tablet 3a03f8a32da8499dbc0e2a426d214001 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251025 20:03:39.957356 30664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a03f8a32da8499dbc0e2a426d214001. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.958566 30738 tablet_service.cc:1505] Processing CreateTablet for tablet d986337b8c2047b4a1931044eb9cc540 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251025 20:03:39.960223 30738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d986337b8c2047b4a1931044eb9cc540. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.967131 30793 tablet_bootstrap.cc:492] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:39.967898 30793 ts_tablet_manager.cc:1403] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.005s
I20251025 20:03:39.967567 30663 tablet_service.cc:1505] Processing CreateTablet for tablet d986337b8c2047b4a1931044eb9cc540 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251025 20:03:39.968525 30737 tablet_service.cc:1505] Processing CreateTablet for tablet d4fa51030cac4f9ba59bc010c9eb0034 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251025 20:03:39.969220 30663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d986337b8c2047b4a1931044eb9cc540. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.970058 30737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4fa51030cac4f9ba59bc010c9eb0034. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.975831 30662 tablet_service.cc:1505] Processing CreateTablet for tablet d4fa51030cac4f9ba59bc010c9eb0034 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251025 20:03:39.976069 30661 tablet_service.cc:1505] Processing CreateTablet for tablet 380df7888a29498d99ec3fe64410a542 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251025 20:03:39.977610 30661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 380df7888a29498d99ec3fe64410a542. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.981629 30662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4fa51030cac4f9ba59bc010c9eb0034. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.984126 30793 raft_consensus.cc:359] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:39.987756 30793 raft_consensus.cc:385] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:39.988250 30793 raft_consensus.cc:740] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:39.989310 30793 consensus_queue.cc:260] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:39.996239 30743 tablet_service.cc:1505] Processing CreateTablet for tablet 380df7888a29498d99ec3fe64410a542 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251025 20:03:39.997530 30736 tablet_service.cc:1505] Processing CreateTablet for tablet 27396efdc31841b68b8b2ea7ca531514 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251025 20:03:39.998154 30743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 380df7888a29498d99ec3fe64410a542. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.999135 30736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27396efdc31841b68b8b2ea7ca531514. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:39.999665 30660 tablet_service.cc:1505] Processing CreateTablet for tablet 27396efdc31841b68b8b2ea7ca531514 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251025 20:03:40.001076 30660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27396efdc31841b68b8b2ea7ca531514. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.008651 30735 tablet_service.cc:1505] Processing CreateTablet for tablet 375e703ada5b4ee2a37ea5e2a9c75376 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251025 20:03:40.010571 30735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 375e703ada5b4ee2a37ea5e2a9c75376. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.015324 30734 tablet_service.cc:1505] Processing CreateTablet for tablet d33d554021204714a7a71332612e2070 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251025 20:03:40.017113 30734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d33d554021204714a7a71332612e2070. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.058257 30796 tablet_bootstrap.cc:492] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:40.060247 30663 tablet_service.cc:1505] Processing CreateTablet for tablet a7d77f4dd14642b7bdb7299e84673c03 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251025 20:03:40.068514 30663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d77f4dd14642b7bdb7299e84673c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.087764 30778 heartbeater.cc:499] Master 127.28.192.254:41145 was elected leader, sending a full tablet report...
I20251025 20:03:40.094566 30742 tablet_service.cc:1505] Processing CreateTablet for tablet 97f7928c55054a2f809df6e4ee99800a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251025 20:03:40.095170 30793 ts_tablet_manager.cc:1434] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.114s user 0.016s sys 0.004s
I20251025 20:03:40.100121 30737 tablet_service.cc:1505] Processing CreateTablet for tablet 5e47b11c8d47461cac220a7cc3625f58 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251025 20:03:40.106488 30735 tablet_service.cc:1505] Processing CreateTablet for tablet 0d23f4ba1a4b411baf57f59f30489265 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251025 20:03:40.108186 30734 tablet_service.cc:1505] Processing CreateTablet for tablet 20fe84e124304a7a9d92910cfa619beb (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251025 20:03:40.111075 30796 tablet_bootstrap.cc:654] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:40.113839 30665 tablet_service.cc:1505] Processing CreateTablet for tablet 375e703ada5b4ee2a37ea5e2a9c75376 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251025 20:03:40.129927 30665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 375e703ada5b4ee2a37ea5e2a9c75376. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.132943 30741 tablet_service.cc:1505] Processing CreateTablet for tablet a7d77f4dd14642b7bdb7299e84673c03 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251025 20:03:40.135571 30738 tablet_service.cc:1505] Processing CreateTablet for tablet 54d399a886ee4166a2ea1b4410f9425a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251025 20:03:40.136042 30594 tablet_service.cc:1505] Processing CreateTablet for tablet 78afc6e697b1482391aa5245bee3d881 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251025 20:03:40.137408 30664 tablet_service.cc:1505] Processing CreateTablet for tablet 0d23f4ba1a4b411baf57f59f30489265 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251025 20:03:40.125222 30740 tablet_service.cc:1505] Processing CreateTablet for tablet d1f9f57422164e5182f82b532e1bef47 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251025 20:03:40.138862 30664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d23f4ba1a4b411baf57f59f30489265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.140101 30666 tablet_service.cc:1505] Processing CreateTablet for tablet d33d554021204714a7a71332612e2070 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251025 20:03:40.140473 30593 tablet_service.cc:1505] Processing CreateTablet for tablet 13be59a31cda407db6fa50a400f37e1f (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251025 20:03:40.141381 30666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d33d554021204714a7a71332612e2070. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.126343 30793 tablet_bootstrap.cc:492] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:40.137418 30594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78afc6e697b1482391aa5245bee3d881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.145221 30661 tablet_service.cc:1505] Processing CreateTablet for tablet 20fe84e124304a7a9d92910cfa619beb (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251025 20:03:40.146623 30661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20fe84e124304a7a9d92910cfa619beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.119930 30743 tablet_service.cc:1505] Processing CreateTablet for tablet 531b2bbad426413898122c589cfa9780 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251025 20:03:40.127331 30735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d23f4ba1a4b411baf57f59f30489265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.125805 30660 tablet_service.cc:1505] Processing CreateTablet for tablet 97f7928c55054a2f809df6e4ee99800a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251025 20:03:40.149788 30742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f7928c55054a2f809df6e4ee99800a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.152626 30668 tablet_service.cc:1505] Processing CreateTablet for tablet d1f9f57422164e5182f82b532e1bef47 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251025 20:03:40.160657 30734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20fe84e124304a7a9d92910cfa619beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.149088 30667 tablet_service.cc:1505] Processing CreateTablet for tablet 531b2bbad426413898122c589cfa9780 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251025 20:03:40.147611 30593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13be59a31cda407db6fa50a400f37e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.151546 30669 tablet_service.cc:1505] Processing CreateTablet for tablet 827742f9e2064e96a4fdcbd6a9b9a4a2 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251025 20:03:40.162515 30590 tablet_service.cc:1505] Processing CreateTablet for tablet b406b18ee0364eb5b4e35826fe7290be (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251025 20:03:40.163949 30590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b406b18ee0364eb5b4e35826fe7290be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.167228 30737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e47b11c8d47461cac220a7cc3625f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.168111 30591 tablet_service.cc:1505] Processing CreateTablet for tablet 06d7fa7a698c4890af235aed97fd6fb5 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251025 20:03:40.168767 30589 tablet_service.cc:1505] Processing CreateTablet for tablet 3a03f8a32da8499dbc0e2a426d214001 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251025 20:03:40.171366 30592 tablet_service.cc:1505] Processing CreateTablet for tablet da4360132dff42a18667801272f14797 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251025 20:03:40.171979 30588 tablet_service.cc:1505] Processing CreateTablet for tablet d986337b8c2047b4a1931044eb9cc540 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251025 20:03:40.169709 30591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d7fa7a698c4890af235aed97fd6fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.173434 30588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d986337b8c2047b4a1931044eb9cc540. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.174114 30740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f9f57422164e5182f82b532e1bef47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.174255 30589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a03f8a32da8499dbc0e2a426d214001. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.174999 30592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4360132dff42a18667801272f14797. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.178122 30660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f7928c55054a2f809df6e4ee99800a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.179044 30668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f9f57422164e5182f82b532e1bef47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.182354 30587 tablet_service.cc:1505] Processing CreateTablet for tablet d4fa51030cac4f9ba59bc010c9eb0034 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251025 20:03:40.184011 30587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4fa51030cac4f9ba59bc010c9eb0034. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.149464 30739 tablet_service.cc:1505] Processing CreateTablet for tablet 6cbb781b2e4441b4be9b53f7c0947525 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251025 20:03:40.186352 30738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d399a886ee4166a2ea1b4410f9425a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.196413 30586 tablet_service.cc:1505] Processing CreateTablet for tablet 380df7888a29498d99ec3fe64410a542 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251025 20:03:40.140101 30662 tablet_service.cc:1505] Processing CreateTablet for tablet 5e47b11c8d47461cac220a7cc3625f58 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251025 20:03:40.198155 30586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 380df7888a29498d99ec3fe64410a542. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.206139 30741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d77f4dd14642b7bdb7299e84673c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.207181 30795 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:40.207731 30795 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:40.208581 30796 tablet_bootstrap.cc:492] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:40.209137 30796 ts_tablet_manager.cc:1403] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.155s user 0.013s sys 0.008s
I20251025 20:03:40.210238 30743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 531b2bbad426413898122c589cfa9780. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.212018 30585 tablet_service.cc:1505] Processing CreateTablet for tablet 27396efdc31841b68b8b2ea7ca531514 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251025 20:03:40.213565 30585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27396efdc31841b68b8b2ea7ca531514. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.215734 30795 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:40.216665 30796 raft_consensus.cc:359] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.217751 30796 raft_consensus.cc:385] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:40.220867 30667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 531b2bbad426413898122c589cfa9780. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.218333 30739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cbb781b2e4441b4be9b53f7c0947525. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.231073 30663 tablet_service.cc:1505] Processing CreateTablet for tablet 4f2c3828d3614c1b97d3daf7d33c6537 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
W20251025 20:03:40.246935 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:40.253813 30793 tablet_bootstrap.cc:654] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:40.256184 30663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2c3828d3614c1b97d3daf7d33c6537. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.267045 30660 tablet_service.cc:1505] Processing CreateTablet for tablet 54d399a886ee4166a2ea1b4410f9425a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251025 20:03:40.239006 30796 raft_consensus.cc:740] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:40.294870 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:40.289738 30594 tablet_service.cc:1505] Processing CreateTablet for tablet 375e703ada5b4ee2a37ea5e2a9c75376 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251025 20:03:40.303246 30743 tablet_service.cc:1505] Processing CreateTablet for tablet 5f27e13dcf894a1487ed057731df6174 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251025 20:03:40.312269 30594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 375e703ada5b4ee2a37ea5e2a9c75376. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.314054 30734 tablet_service.cc:1505] Processing CreateTablet for tablet 53eca06f855d4e699f10ab84a2ba0981 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251025 20:03:40.315171 30743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f27e13dcf894a1487ed057731df6174. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.319998 30735 tablet_service.cc:1505] Processing CreateTablet for tablet 542d67f7d7544649be29769edeff1d0b (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251025 20:03:40.317173 30588 tablet_service.cc:1505] Processing CreateTablet for tablet 0d23f4ba1a4b411baf57f59f30489265 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251025 20:03:40.321436 30735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 542d67f7d7544649be29769edeff1d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.321604 30588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d23f4ba1a4b411baf57f59f30489265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.306463 30669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827742f9e2064e96a4fdcbd6a9b9a4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.322830 30734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53eca06f855d4e699f10ab84a2ba0981. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.330128 30660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d399a886ee4166a2ea1b4410f9425a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 20:03:40.331048 30716 leader_election.cc:343] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:40.125222 30736 tablet_service.cc:1505] Processing CreateTablet for tablet 827742f9e2064e96a4fdcbd6a9b9a4a2 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251025 20:03:40.334606 30741 tablet_service.cc:1505] Processing CreateTablet for tablet 4f2c3828d3614c1b97d3daf7d33c6537 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251025 20:03:40.343861 30665 tablet_service.cc:1505] Processing CreateTablet for tablet 6cbb781b2e4441b4be9b53f7c0947525 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251025 20:03:40.345115 30741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2c3828d3614c1b97d3daf7d33c6537. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.344050 30593 tablet_service.cc:1505] Processing CreateTablet for tablet 97f7928c55054a2f809df6e4ee99800a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251025 20:03:40.348034 30661 tablet_service.cc:1505] Processing CreateTablet for tablet 53eca06f855d4e699f10ab84a2ba0981 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251025 20:03:40.331446 30737 tablet_service.cc:1505] Processing CreateTablet for tablet 2d7ed8359ea1484ca7ff724d58fa7a87 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251025 20:03:40.354583 30662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e47b11c8d47461cac220a7cc3625f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.374511 30666 tablet_service.cc:1505] Processing CreateTablet for tablet 5f27e13dcf894a1487ed057731df6174 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251025 20:03:40.376044 30666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f27e13dcf894a1487ed057731df6174. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.332113 30592 tablet_service.cc:1505] Processing CreateTablet for tablet 20fe84e124304a7a9d92910cfa619beb (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251025 20:03:40.367929 30586 tablet_service.cc:1505] Processing CreateTablet for tablet d33d554021204714a7a71332612e2070 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251025 20:03:40.389236 30585 tablet_service.cc:1505] Processing CreateTablet for tablet 5e47b11c8d47461cac220a7cc3625f58 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251025 20:03:40.389236 30591 tablet_service.cc:1505] Processing CreateTablet for tablet 531b2bbad426413898122c589cfa9780 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251025 20:03:40.357738 30796 consensus_queue.cc:260] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.385039 30736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827742f9e2064e96a4fdcbd6a9b9a4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.406232 30737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d7ed8359ea1484ca7ff724d58fa7a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.400434 30593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f7928c55054a2f809df6e4ee99800a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.412577 30591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 531b2bbad426413898122c589cfa9780. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.401031 30665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cbb781b2e4441b4be9b53f7c0947525. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.415175 30585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e47b11c8d47461cac220a7cc3625f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.406802 30800 tablet_bootstrap.cc:492] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:40.418399 30586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d33d554021204714a7a71332612e2070. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.389650 30742 tablet_service.cc:1505] Processing CreateTablet for tablet bc001e0fba604d68bfa98fc25d70bbbb (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251025 20:03:40.429668 30742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc001e0fba604d68bfa98fc25d70bbbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.437712 30661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53eca06f855d4e699f10ab84a2ba0981. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.445654 30629 heartbeater.cc:499] Master 127.28.192.254:41145 was elected leader, sending a full tablet report...
I20251025 20:03:40.451637 30590 tablet_service.cc:1505] Processing CreateTablet for tablet 827742f9e2064e96a4fdcbd6a9b9a4a2 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251025 20:03:40.462926 30800 tablet_bootstrap.cc:654] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:40.473641 30661 tablet_service.cc:1505] Processing CreateTablet for tablet 542d67f7d7544649be29769edeff1d0b (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251025 20:03:40.475071 30661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 542d67f7d7544649be29769edeff1d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.475332 30589 tablet_service.cc:1505] Processing CreateTablet for tablet d1f9f57422164e5182f82b532e1bef47 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251025 20:03:40.477416 30664 tablet_service.cc:1505] Processing CreateTablet for tablet 2d7ed8359ea1484ca7ff724d58fa7a87 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251025 20:03:40.478863 30664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d7ed8359ea1484ca7ff724d58fa7a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.480418 30796 ts_tablet_manager.cc:1434] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.271s user 0.012s sys 0.004s
I20251025 20:03:40.481251 30704 heartbeater.cc:499] Master 127.28.192.254:41145 was elected leader, sending a full tablet report...
I20251025 20:03:40.481530 30796 tablet_bootstrap.cc:492] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:40.482023 30592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20fe84e124304a7a9d92910cfa619beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.501811 30738 tablet_service.cc:1505] Processing CreateTablet for tablet e4bb9e40b009470eb783f9bf63e0c3ba (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251025 20:03:40.503571 30738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4bb9e40b009470eb783f9bf63e0c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.481629 30669 tablet_service.cc:1505] Processing CreateTablet for tablet ec9d5d5c538f4253bcec6b8b23843ed6 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251025 20:03:40.493649 30739 tablet_service.cc:1505] Processing CreateTablet for tablet ec9d5d5c538f4253bcec6b8b23843ed6 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251025 20:03:40.523844 30739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec9d5d5c538f4253bcec6b8b23843ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.529124 30591 tablet_service.cc:1505] Processing CreateTablet for tablet 54d399a886ee4166a2ea1b4410f9425a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251025 20:03:40.538162 30589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f9f57422164e5182f82b532e1bef47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.538851 30590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827742f9e2064e96a4fdcbd6a9b9a4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.484843 30740 tablet_service.cc:1505] Processing CreateTablet for tablet 76be6727aa3a42939ff2cd45105b91f3 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251025 20:03:40.544018 30740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76be6727aa3a42939ff2cd45105b91f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.552769 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:40.554360 30585 tablet_service.cc:1505] Processing CreateTablet for tablet a7d77f4dd14642b7bdb7299e84673c03 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251025 20:03:40.554555 30668 tablet_service.cc:1505] Processing CreateTablet for tablet 76be6727aa3a42939ff2cd45105b91f3 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251025 20:03:40.555817 30585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d77f4dd14642b7bdb7299e84673c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.558568 30734 tablet_service.cc:1505] Processing CreateTablet for tablet b0a9871641274e83b2300f3ad8095add (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251025 20:03:40.559991 30734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a9871641274e83b2300f3ad8095add. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.519999 30669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec9d5d5c538f4253bcec6b8b23843ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.566349 30591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d399a886ee4166a2ea1b4410f9425a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.569188 30587 tablet_service.cc:1505] Processing CreateTablet for tablet 6cbb781b2e4441b4be9b53f7c0947525 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251025 20:03:40.553717 30737 tablet_service.cc:1505] Processing CreateTablet for tablet 7904fa0f61dd4c9fadc39faddc8f908c (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251025 20:03:40.573081 30737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7904fa0f61dd4c9fadc39faddc8f908c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.574354 30800 tablet_bootstrap.cc:492] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:40.574904 30800 ts_tablet_manager.cc:1403] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.174s user 0.025s sys 0.017s
I20251025 20:03:40.584285 30793 tablet_bootstrap.cc:492] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:40.584904 30793 ts_tablet_manager.cc:1403] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.459s user 0.029s sys 0.034s
I20251025 20:03:40.588197 30793 raft_consensus.cc:359] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.588995 30793 raft_consensus.cc:385] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:40.589305 30793 raft_consensus.cc:740] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:40.590166 30793 consensus_queue.cc:260] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.594115 30668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76be6727aa3a42939ff2cd45105b91f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.596057 30793 ts_tablet_manager.cc:1434] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20251025 20:03:40.597256 30793 tablet_bootstrap.cc:492] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:40.604480 30663 tablet_service.cc:1505] Processing CreateTablet for tablet e4bb9e40b009470eb783f9bf63e0c3ba (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251025 20:03:40.606032 30663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4bb9e40b009470eb783f9bf63e0c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 20:03:40.611146 30719 leader_election.cc:343] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:40.611701 30719 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:40.612779 30795 raft_consensus.cc:2749] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:40.613152 30796 tablet_bootstrap.cc:654] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:40.618757 30587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cbb781b2e4441b4be9b53f7c0947525. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.620857 30594 tablet_service.cc:1505] Processing CreateTablet for tablet db5a3b726f79432baf8b25596709ca04 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251025 20:03:40.622438 30594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db5a3b726f79432baf8b25596709ca04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.625986 30741 tablet_service.cc:1505] Processing CreateTablet for tablet 813ab494bf654127a41723bb256c1ede (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251025 20:03:40.627745 30741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813ab494bf654127a41723bb256c1ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.636833 30743 tablet_service.cc:1505] Processing CreateTablet for tablet d539e97d6aea48aeb11438d95241d1bd (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251025 20:03:40.638760 30743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d539e97d6aea48aeb11438d95241d1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.639678 30800 raft_consensus.cc:359] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.640545 30800 raft_consensus.cc:385] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:40.640933 30800 raft_consensus.cc:740] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:40.643704 30586 tablet_service.cc:1505] Processing CreateTablet for tablet 53eca06f855d4e699f10ab84a2ba0981 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251025 20:03:40.645634 30586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53eca06f855d4e699f10ab84a2ba0981. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.648949 30591 tablet_service.cc:1505] Processing CreateTablet for tablet 1d5b049633ac4f21b7a8e34d536fd164 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251025 20:03:40.663806 30591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d5b049633ac4f21b7a8e34d536fd164. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.671658 30802 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:40.672276 30802 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.677562 30802 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:40.678927 30793 tablet_bootstrap.cc:654] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:40.661725 30737 tablet_service.cc:1505] Processing CreateTablet for tablet dc26ecef9f79461a932461d9c32a094f (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251025 20:03:40.683328 30737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc26ecef9f79461a932461d9c32a094f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.687776 30663 tablet_service.cc:1505] Processing CreateTablet for tablet b0a9871641274e83b2300f3ad8095add (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251025 20:03:40.691337 30663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a9871641274e83b2300f3ad8095add. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.700656 30588 tablet_service.cc:1505] Processing CreateTablet for tablet 5f27e13dcf894a1487ed057731df6174 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251025 20:03:40.702883 30793 tablet_bootstrap.cc:492] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:40.703482 30793 ts_tablet_manager.cc:1403] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.106s user 0.014s sys 0.005s
I20251025 20:03:40.589344 30667 tablet_service.cc:1505] Processing CreateTablet for tablet bc001e0fba604d68bfa98fc25d70bbbb (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251025 20:03:40.711220 30793 raft_consensus.cc:359] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.641692 30800 consensus_queue.cc:260] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.712059 30793 raft_consensus.cc:385] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:40.713454 30667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc001e0fba604d68bfa98fc25d70bbbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.712308 30663 tablet_service.cc:1505] Processing CreateTablet for tablet d539e97d6aea48aeb11438d95241d1bd (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251025 20:03:40.712417 30793 raft_consensus.cc:740] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:40.716554 30663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d539e97d6aea48aeb11438d95241d1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.716464 30793 consensus_queue.cc:260] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.718641 30735 tablet_service.cc:1505] Processing CreateTablet for tablet 6973d3401da04486853abdd647a5795a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251025 20:03:40.720408 30735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6973d3401da04486853abdd647a5795a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.709869 30660 tablet_service.cc:1505] Processing CreateTablet for tablet 813ab494bf654127a41723bb256c1ede (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251025 20:03:40.732146 30738 tablet_service.cc:1505] Processing CreateTablet for tablet 9cb15f2debdf450ab73ed74f6633fea7 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251025 20:03:40.733301 30660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813ab494bf654127a41723bb256c1ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.741779 30662 tablet_service.cc:1505] Processing CreateTablet for tablet dc26ecef9f79461a932461d9c32a094f (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251025 20:03:40.743863 30662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc26ecef9f79461a932461d9c32a094f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.746626 30588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f27e13dcf894a1487ed057731df6174. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.751291 30665 tablet_service.cc:1505] Processing CreateTablet for tablet 7904fa0f61dd4c9fadc39faddc8f908c (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251025 20:03:40.753227 30665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7904fa0f61dd4c9fadc39faddc8f908c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.759080 30796 tablet_bootstrap.cc:492] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:40.759696 30796 ts_tablet_manager.cc:1403] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.278s user 0.011s sys 0.007s
I20251025 20:03:40.779531 30661 tablet_service.cc:1505] Processing CreateTablet for tablet 6973d3401da04486853abdd647a5795a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251025 20:03:40.781590 30661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6973d3401da04486853abdd647a5795a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.782513 30796 raft_consensus.cc:359] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.866250 30796 raft_consensus.cc:385] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:40.866619 30796 raft_consensus.cc:740] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:40.867411 30796 consensus_queue.cc:260] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.733392 30738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb15f2debdf450ab73ed74f6633fea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.813823 30793 ts_tablet_manager.cc:1434] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.110s user 0.008s sys 0.012s
I20251025 20:03:40.823246 30594 tablet_service.cc:1505] Processing CreateTablet for tablet 4f2c3828d3614c1b97d3daf7d33c6537 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251025 20:03:40.871075 30793 tablet_bootstrap.cc:492] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:40.871955 30594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2c3828d3614c1b97d3daf7d33c6537. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.875602 30796 ts_tablet_manager.cc:1434] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.110s user 0.008s sys 0.001s
I20251025 20:03:40.832765 30667 tablet_service.cc:1505] Processing CreateTablet for tablet 393c7e0e9757478198789a4dd6c9a8a8 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251025 20:03:40.825656 30663 tablet_service.cc:1505] Processing CreateTablet for tablet 719d97d4d3534235ad57ba5d3d50de74 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
W20251025 20:03:40.834233 30489 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 121570us
I20251025 20:03:40.835024 30591 tablet_service.cc:1505] Processing CreateTablet for tablet 542d67f7d7544649be29769edeff1d0b (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251025 20:03:40.836866 30590 tablet_service.cc:1505] Processing CreateTablet for tablet 2d7ed8359ea1484ca7ff724d58fa7a87 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251025 20:03:40.836879 30737 tablet_service.cc:1505] Processing CreateTablet for tablet d4af0795c8ff48d9b639058ea7111f12 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251025 20:03:40.887239 30591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 542d67f7d7544649be29769edeff1d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.842257 30742 tablet_service.cc:1505] Processing CreateTablet for tablet 1df036864f934f2d81add520300e9ed6 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251025 20:03:40.849884 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:40.889705 30645 leader_election.cc:343] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:40.890492 30590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d7ed8359ea1484ca7ff724d58fa7a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.833545 30739 tablet_service.cc:1505] Processing CreateTablet for tablet 393c7e0e9757478198789a4dd6c9a8a8 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251025 20:03:40.895275 30669 tablet_service.cc:1505] Processing CreateTablet for tablet 9cb15f2debdf450ab73ed74f6633fea7 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251025 20:03:40.900153 30807 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:40.900700 30807 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.863855 30735 tablet_service.cc:1505] Processing CreateTablet for tablet db5a3b726f79432baf8b25596709ca04 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251025 20:03:40.828015 30736 tablet_service.cc:1505] Processing CreateTablet for tablet 719d97d4d3534235ad57ba5d3d50de74 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251025 20:03:40.871012 30740 tablet_service.cc:1505] Processing CreateTablet for tablet 4403a496a85d4a3ebab45b2cb1af580e (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251025 20:03:40.878943 30793 tablet_bootstrap.cc:654] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:40.906659 30807 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:40.842070 30593 tablet_service.cc:1505] Processing CreateTablet for tablet ec9d5d5c538f4253bcec6b8b23843ed6 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251025 20:03:40.887358 30737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4af0795c8ff48d9b639058ea7111f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.845309 30800 ts_tablet_manager.cc:1434] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.270s user 0.011s sys 0.004s
I20251025 20:03:40.912249 30585 tablet_service.cc:1505] Processing CreateTablet for tablet 76be6727aa3a42939ff2cd45105b91f3 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251025 20:03:40.919976 30663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719d97d4d3534235ad57ba5d3d50de74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.920389 30589 tablet_service.cc:1505] Processing CreateTablet for tablet bc001e0fba604d68bfa98fc25d70bbbb (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251025 20:03:40.922273 30593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec9d5d5c538f4253bcec6b8b23843ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.926812 30802 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:40.927366 30802 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.927529 30668 tablet_service.cc:1505] Processing CreateTablet for tablet 1df036864f934f2d81add520300e9ed6 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251025 20:03:40.928934 30668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df036864f934f2d81add520300e9ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.929414 30802 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:40.930800 30796 tablet_bootstrap.cc:492] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:40.931360 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:40.931671 30738 tablet_service.cc:1505] Processing CreateTablet for tablet 1d5b049633ac4f21b7a8e34d536fd164 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
W20251025 20:03:40.933277 30645 leader_election.cc:343] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:40.941098 30664 tablet_service.cc:1505] Processing CreateTablet for tablet db5a3b726f79432baf8b25596709ca04 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251025 20:03:40.941771 30589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc001e0fba604d68bfa98fc25d70bbbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.943107 30800 tablet_bootstrap.cc:492] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:40.946697 30585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76be6727aa3a42939ff2cd45105b91f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.968214 30587 tablet_service.cc:1505] Processing CreateTablet for tablet e4bb9e40b009470eb783f9bf63e0c3ba (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251025 20:03:40.973999 30736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719d97d4d3534235ad57ba5d3d50de74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.974280 30669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb15f2debdf450ab73ed74f6633fea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.975617 30667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 393c7e0e9757478198789a4dd6c9a8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.977393 30666 tablet_service.cc:1505] Processing CreateTablet for tablet 4403a496a85d4a3ebab45b2cb1af580e (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251025 20:03:40.978228 30734 tablet_service.cc:1505] Processing CreateTablet for tablet b29cc29499b448adb358d6e30d3be8aa (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251025 20:03:40.978843 30666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4403a496a85d4a3ebab45b2cb1af580e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.978237 30735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db5a3b726f79432baf8b25596709ca04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.980640 30664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db5a3b726f79432baf8b25596709ca04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.982208 30740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4403a496a85d4a3ebab45b2cb1af580e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.926813 30662 tablet_service.cc:1505] Processing CreateTablet for tablet d4af0795c8ff48d9b639058ea7111f12 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251025 20:03:40.984632 30739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 393c7e0e9757478198789a4dd6c9a8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.985040 30662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4af0795c8ff48d9b639058ea7111f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.985767 30795 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:40.986218 30738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d5b049633ac4f21b7a8e34d536fd164. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.986300 30795 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:40.988590 30795 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:40.989840 30734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29cc29499b448adb358d6e30d3be8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:40.969765 30587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4bb9e40b009470eb783f9bf63e0c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.005615 30796 tablet_bootstrap.cc:654] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.007586 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:41.025530 30594 tablet_service.cc:1505] Processing CreateTablet for tablet 813ab494bf654127a41723bb256c1ede (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251025 20:03:41.008572 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:41.008828 30795 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:41.051216 30594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813ab494bf654127a41723bb256c1ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.051329 30795 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.010385 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:41.010334 30591 tablet_service.cc:1505] Processing CreateTablet for tablet b0a9871641274e83b2300f3ad8095add (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251025 20:03:41.011168 30793 tablet_bootstrap.cc:492] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:41.053555 30793 ts_tablet_manager.cc:1403] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.183s user 0.035s sys 0.003s
I20251025 20:03:41.054620 30795 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:41.054986 30665 tablet_service.cc:1505] Processing CreateTablet for tablet 1d5b049633ac4f21b7a8e34d536fd164 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251025 20:03:41.056492 30665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d5b049633ac4f21b7a8e34d536fd164. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.056879 30793 raft_consensus.cc:359] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.057413 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:41.058195 30716 leader_election.cc:343] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.019644 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:41.032763 30592 tablet_service.cc:1505] Processing CreateTablet for tablet dc26ecef9f79461a932461d9c32a094f (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
W20251025 20:03:41.059770 30716 leader_election.cc:343] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:41.053471 30645 leader_election.cc:343] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.059948 30592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc26ecef9f79461a932461d9c32a094f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 20:03:41.060197 30719 leader_election.cc:343] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.060379 30645 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:41.060724 30719 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:03:41.062003 30645 leader_election.cc:343] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.062264 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:41.062448 30645 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:41.043128 30586 tablet_service.cc:1505] Processing CreateTablet for tablet d539e97d6aea48aeb11438d95241d1bd (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251025 20:03:41.037278 30588 tablet_service.cc:1505] Processing CreateTablet for tablet 6973d3401da04486853abdd647a5795a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251025 20:03:41.023679 30587 tablet_service.cc:1505] Processing CreateTablet for tablet 7904fa0f61dd4c9fadc39faddc8f908c (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251025 20:03:41.057732 30793 raft_consensus.cc:385] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.065308 30802 raft_consensus.cc:2749] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:41.068040 30802 raft_consensus.cc:2749] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 20:03:41.067080 30719 leader_election.cc:343] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.068904 30719 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:41.064630 30795 raft_consensus.cc:2749] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:41.052959 30742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df036864f934f2d81add520300e9ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.069854 30793 raft_consensus.cc:740] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.075186 30795 raft_consensus.cc:2749] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:41.075946 30793 consensus_queue.cc:260] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.078534 30793 ts_tablet_manager.cc:1434] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20251025 20:03:41.079631 30793 tablet_bootstrap.cc:492] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:41.073210 30800 tablet_bootstrap.cc:654] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.097249 30741 tablet_service.cc:1505] Processing CreateTablet for tablet 775d2ddde3b44056818240b2a8402e9c (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251025 20:03:41.099042 30741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 775d2ddde3b44056818240b2a8402e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.102555 30662 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 40) took 1016 ms. Trace:
I20251025 20:03:41.103636 30793 tablet_bootstrap.cc:654] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.103693 30660 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 33) took 1020 ms. Trace:
I20251025 20:03:41.103148 30662 rpcz_store.cc:276] 1025 20:03:40.121780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.925748 (+803968us) service_pool.cc:225] Handling call
1025 20:03:40.984552 (+ 58804us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.984666 (+ 114us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.034669 (+ 50003us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.093683 (+ 59014us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.102468 (+ 8785us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.104295 30660 rpcz_store.cc:276] 1025 20:03:40.083582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.708843 (+625261us) service_pool.cc:225] Handling call
1025 20:03:40.732734 (+ 23891us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.732846 (+ 112us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.034824 (+301978us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.103136 (+ 68312us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.103611 (+ 475us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.106489 30660 tablet_service.cc:1505] Processing CreateTablet for tablet b29cc29499b448adb358d6e30d3be8aa (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251025 20:03:41.107988 30660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29cc29499b448adb358d6e30d3be8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.112124 30585 tablet_service.cc:1505] Processing CreateTablet for tablet 9cb15f2debdf450ab73ed74f6633fea7 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251025 20:03:41.113278 30743 tablet_service.cc:1505] Processing CreateTablet for tablet 424773f793a4443da2db316842a1f3b5 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251025 20:03:41.106492 30662 tablet_service.cc:1505] Processing CreateTablet for tablet 775d2ddde3b44056818240b2a8402e9c (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251025 20:03:41.114717 30743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 424773f793a4443da2db316842a1f3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.115321 30662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 775d2ddde3b44056818240b2a8402e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.139556 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:41.140362 30753 raft_consensus.cc:2468] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:41.141868 30570 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:41.142899 30807 raft_consensus.cc:2804] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:41.143288 30807 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:41.143628 30807 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:41.144804 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:41.147231 30567 leader_election.cc:343] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.149168 30668 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 42) took 1026 ms. Trace:
W20251025 20:03:41.150174 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 83132us
I20251025 20:03:41.156699 30737 tablet_service.cc:1505] Processing CreateTablet for tablet fea0c74c9c0c4a69afe072c45c4cbe99 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251025 20:03:41.158217 30737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea0c74c9c0c4a69afe072c45c4cbe99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.127162 30591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a9871641274e83b2300f3ad8095add. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.166301 30587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7904fa0f61dd4c9fadc39faddc8f908c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.168040 30662 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 46) took 1044 ms. Trace:
I20251025 20:03:41.169440 30585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb15f2debdf450ab73ed74f6633fea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.168725 30662 rpcz_store.cc:276] 1025 20:03:40.124100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.105426 (+981326us) service_pool.cc:225] Handling call
1025 20:03:41.114864 (+ 9438us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.114956 (+ 92us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.154832 (+ 39876us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.167260 (+ 12428us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.167943 (+ 683us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.170226 30588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6973d3401da04486853abdd647a5795a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.171900 30662 tablet_service.cc:1505] Processing CreateTablet for tablet 424773f793a4443da2db316842a1f3b5 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251025 20:03:41.120188 30661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 36) took 1035 ms. Trace:
I20251025 20:03:41.174021 30662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 424773f793a4443da2db316842a1f3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.177754 30738 tablet_service.cc:1505] Processing CreateTablet for tablet e8c971a6d4854180b4d243fa79fbbc11 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251025 20:03:41.149607 30668 rpcz_store.cc:276] 1025 20:03:40.122656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.926428 (+803772us) service_pool.cc:225] Handling call
1025 20:03:40.928420 (+ 1992us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.928521 (+ 101us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.023436 (+ 94915us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.148583 (+125147us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.149101 (+ 518us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.186735 30668 tablet_service.cc:1505] Processing CreateTablet for tablet e8c971a6d4854180b4d243fa79fbbc11 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251025 20:03:41.183897 30738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c971a6d4854180b4d243fa79fbbc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.184609 30593 tablet_service.cc:1505] Processing CreateTablet for tablet 719d97d4d3534235ad57ba5d3d50de74 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251025 20:03:41.190203 30590 tablet_service.cc:1505] Processing CreateTablet for tablet 393c7e0e9757478198789a4dd6c9a8a8 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251025 20:03:41.191691 30793 tablet_bootstrap.cc:492] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:41.192225 30793 ts_tablet_manager.cc:1403] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.113s user 0.033s sys 0.029s
I20251025 20:03:41.195390 30793 raft_consensus.cc:359] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.196193 30793 raft_consensus.cc:385] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.196527 30793 raft_consensus.cc:740] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.201367 30662 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 47) took 1077 ms. Trace:
I20251025 20:03:41.201885 30662 rpcz_store.cc:276] 1025 20:03:40.124510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.170348 (+1045838us) service_pool.cc:225] Handling call
1025 20:03:41.173290 (+ 2942us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.173445 (+ 155us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.192888 (+ 19443us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.200793 (+ 7905us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.201297 (+ 504us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.204198 30666 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 41) took 1082 ms. Trace:
I20251025 20:03:41.204069 30662 tablet_service.cc:1505] Processing CreateTablet for tablet fea0c74c9c0c4a69afe072c45c4cbe99 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251025 20:03:41.204722 30666 rpcz_store.cc:276] 1025 20:03:40.122314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.925853 (+803539us) service_pool.cc:225] Handling call
1025 20:03:40.978350 (+ 52497us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.978456 (+ 106us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.027649 (+ 49193us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.203599 (+175950us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.204124 (+ 525us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.205677 30662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea0c74c9c0c4a69afe072c45c4cbe99. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 20:03:41.122409 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 60435us
I20251025 20:03:41.185158 30663 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 38) took 1100 ms. Trace:
I20251025 20:03:41.210495 30740 tablet_service.cc:1505] Processing CreateTablet for tablet da8a0eaa68044835a4cf94340ca25ac5 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251025 20:03:41.193591 30669 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 37) took 1108 ms. Trace:
I20251025 20:03:41.210816 30666 tablet_service.cc:1505] Processing CreateTablet for tablet da8a0eaa68044835a4cf94340ca25ac5 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251025 20:03:41.208223 30663 rpcz_store.cc:276] 1025 20:03:40.085233 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.824673 (+739440us) service_pool.cc:225] Handling call
1025 20:03:40.919415 (+ 94742us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.919531 (+ 116us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.080506 (+160975us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.184586 (+104080us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.185088 (+ 502us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.228636 30663 tablet_service.cc:1505] Processing CreateTablet for tablet 2eddbfc136b841098915d8ea2d82956a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251025 20:03:41.211494 30586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d539e97d6aea48aeb11438d95241d1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.230396 30590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 393c7e0e9757478198789a4dd6c9a8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.232501 30593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719d97d4d3534235ad57ba5d3d50de74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.197355 30793 consensus_queue.cc:260] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.234894 30793 ts_tablet_manager.cc:1434] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.042s user 0.009s sys 0.000s
I20251025 20:03:41.236160 30793 tablet_bootstrap.cc:492] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:41.218609 30669 rpcz_store.cc:276] 1025 20:03:40.084880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.821407 (+736527us) service_pool.cc:225] Handling call
1025 20:03:40.917189 (+ 95782us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.918105 (+ 916us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.167940 (+249835us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.193039 (+ 25099us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.193524 (+ 485us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.215729 30668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c971a6d4854180b4d243fa79fbbc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.207751 30807 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.247669 30669 tablet_service.cc:1505] Processing CreateTablet for tablet 8e7a1b97c4aa4812a26b178e32476392 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251025 20:03:41.248996 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:03:41.249483 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:41.249625 30753 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:41.253352 30664 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 43) took 1130 ms. Trace:
I20251025 20:03:41.257311 30594 tablet_service.cc:1505] Processing CreateTablet for tablet d4af0795c8ff48d9b639058ea7111f12 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251025 20:03:41.257818 30664 rpcz_store.cc:276] 1025 20:03:40.123005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.939955 (+816950us) service_pool.cc:225] Handling call
1025 20:03:40.950084 (+ 10129us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.950201 (+ 117us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.044019 (+ 93818us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.229350 (+185331us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.253286 (+ 23936us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.258389 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:41.258805 30793 tablet_bootstrap.cc:654] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.258905 30594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4af0795c8ff48d9b639058ea7111f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.177246 30661 rpcz_store.cc:276] 1025 20:03:40.084552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.778017 (+693465us) service_pool.cc:225] Handling call
1025 20:03:40.780869 (+ 2852us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.781009 (+ 140us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.009561 (+228552us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.081683 (+ 72122us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.119116 (+ 37433us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:03:41.262233 30567 leader_election.cc:343] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.261723 30796 tablet_bootstrap.cc:492] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:41.263087 30796 ts_tablet_manager.cc:1403] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.333s user 0.011s sys 0.006s
I20251025 20:03:41.262701 30661 tablet_service.cc:1505] Processing CreateTablet for tablet 4614bdb7f750431fa840cabccf782b4b (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251025 20:03:41.266543 30796 raft_consensus.cc:359] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.286933 30667 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 39) took 1201 ms. Trace:
I20251025 20:03:41.287426 30667 rpcz_store.cc:276] 1025 20:03:40.085580 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.831776 (+746196us) service_pool.cc:225] Handling call
1025 20:03:40.936558 (+104782us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.942259 (+ 5701us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.161468 (+219209us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.286281 (+124813us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.286846 (+ 565us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.298947 30666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da8a0eaa68044835a4cf94340ca25ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.300094 30815 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:41.300434 30663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eddbfc136b841098915d8ea2d82956a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.300686 30815 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.303151 30815 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:41.297521 30594 tablet_service.cc:1505] Processing CreateTablet for tablet 4403a496a85d4a3ebab45b2cb1af580e (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251025 20:03:41.304461 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:41.304811 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:41.305776 30594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4403a496a85d4a3ebab45b2cb1af580e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 20:03:41.306156 30716 leader_election.cc:343] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.306433 30661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4614bdb7f750431fa840cabccf782b4b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 20:03:41.306802 30719 leader_election.cc:343] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.307274 30719 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:41.308733 30815 raft_consensus.cc:2749] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:41.299036 30660 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 45) took 1175 ms. Trace:
I20251025 20:03:41.309700 30740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da8a0eaa68044835a4cf94340ca25ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.250867 30665 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 44) took 1102 ms. Trace:
I20251025 20:03:41.318564 30735 tablet_service.cc:1505] Processing CreateTablet for tablet 1f781ca57ae9473b858d6d4d6bceeb53 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251025 20:03:41.311908 30665 rpcz_store.cc:276] 1025 20:03:40.123355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.053994 (+930639us) service_pool.cc:225] Handling call
1025 20:03:41.055943 (+ 1949us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.056052 (+ 109us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.101385 (+ 45333us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.225389 (+124004us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.225852 (+ 463us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.320212 30669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7a1b97c4aa4812a26b178e32476392. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.320678 30665 tablet_service.cc:1505] Processing CreateTablet for tablet 43d8dfe478a4430da93c214b20963f9f (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251025 20:03:41.311928 30660 rpcz_store.cc:276] 1025 20:03:40.123734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.105365 (+981631us) service_pool.cc:225] Handling call
1025 20:03:41.107428 (+ 2063us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.107540 (+ 112us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.203853 (+ 96313us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.269823 (+ 65970us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.298932 (+ 29109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.310916 30796 raft_consensus.cc:385] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.324110 30796 raft_consensus.cc:740] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.324520 30660 tablet_service.cc:1505] Processing CreateTablet for tablet 580e75e8dcd949e39aff92ada50b40be (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251025 20:03:41.251169 30807 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:41.321789 30665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d8dfe478a4430da93c214b20963f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.326908 30664 tablet_service.cc:1505] Processing CreateTablet for tablet 3f1269e0f1364a69b9a36ce877b0da5e (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251025 20:03:41.347654 30592 tablet_service.cc:1505] Processing CreateTablet for tablet 1df036864f934f2d81add520300e9ed6 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251025 20:03:41.344851 30793 tablet_bootstrap.cc:492] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:41.348119 30663 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 51) took 1104 ms. Trace:
I20251025 20:03:41.343936 30739 tablet_service.cc:1505] Processing CreateTablet for tablet 8e7a1b97c4aa4812a26b178e32476392 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251025 20:03:41.350370 30800 tablet_bootstrap.cc:492] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:41.348996 30663 rpcz_store.cc:276] 1025 20:03:40.243714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.227607 (+983893us) service_pool.cc:225] Handling call
1025 20:03:41.229563 (+ 1956us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.229689 (+ 126us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.340636 (+110947us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.347584 (+ 6948us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.348051 (+ 467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.352089 30800 ts_tablet_manager.cc:1403] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.409s user 0.024s sys 0.021s
I20251025 20:03:41.353339 30589 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 29) took 1010 ms. Trace:
I20251025 20:03:41.353758 30589 rpcz_store.cc:276] 1025 20:03:40.343063 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.918906 (+575843us) service_pool.cc:225] Handling call
1025 20:03:40.941129 (+ 22223us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.941273 (+ 144us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.101397 (+160124us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.352792 (+251395us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.353271 (+ 479us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3968}
I20251025 20:03:41.354835 30734 tablet_service.cc:1505] Processing CreateTablet for tablet 43d8dfe478a4430da93c214b20963f9f (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251025 20:03:41.353192 30663 tablet_service.cc:1505] Processing CreateTablet for tablet 1f781ca57ae9473b858d6d4d6bceeb53 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251025 20:03:41.357120 30734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d8dfe478a4430da93c214b20963f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.352099 30793 ts_tablet_manager.cc:1403] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.116s user 0.038s sys 0.013s
I20251025 20:03:41.360672 30739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7a1b97c4aa4812a26b178e32476392. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.361218 30793 raft_consensus.cc:359] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.362027 30793 raft_consensus.cc:385] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.362354 30793 raft_consensus.cc:740] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.362687 30589 tablet_service.cc:1505] Processing CreateTablet for tablet b29cc29499b448adb358d6e30d3be8aa (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251025 20:03:41.363305 30660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580e75e8dcd949e39aff92ada50b40be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.364162 30589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29cc29499b448adb358d6e30d3be8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.327117 30736 tablet_service.cc:1505] Processing CreateTablet for tablet 3f1269e0f1364a69b9a36ce877b0da5e (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251025 20:03:41.366194 30663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f781ca57ae9473b858d6d4d6bceeb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.367046 30736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1269e0f1364a69b9a36ce877b0da5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.357153 30800 raft_consensus.cc:359] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:41.378129 30800 raft_consensus.cc:385] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.378535 30800 raft_consensus.cc:740] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.379663 30800 consensus_queue.cc:260] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
W20251025 20:03:41.389160 30779 tablet.cc:2378] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:03:41.292522 30743 tablet_service.cc:1505] Processing CreateTablet for tablet 2eddbfc136b841098915d8ea2d82956a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251025 20:03:41.389945 30800 ts_tablet_manager.cc:1434] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.038s user 0.003s sys 0.005s
I20251025 20:03:41.390822 30743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eddbfc136b841098915d8ea2d82956a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.390759 30793 consensus_queue.cc:260] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.393378 30793 ts_tablet_manager.cc:1434] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.035s user 0.001s sys 0.004s
I20251025 20:03:41.393512 30590 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 39) took 1046 ms. Trace:
I20251025 20:03:41.402066 30590 rpcz_store.cc:276] 1025 20:03:40.346699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.189215 (+842516us) service_pool.cc:225] Handling call
1025 20:03:41.191085 (+ 1870us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.191179 (+ 94us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.315425 (+124246us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.392972 (+ 77547us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.393444 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.410595 30663 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 58) took 1008 ms. Trace:
I20251025 20:03:41.404574 30800 tablet_bootstrap.cc:492] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:41.411000 30663 rpcz_store.cc:276] 1025 20:03:40.402624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.352254 (+949630us) service_pool.cc:225] Handling call
1025 20:03:41.356242 (+ 3988us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.356333 (+ 91us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.402038 (+ 45705us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.410082 (+ 8044us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.410533 (+ 451us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.414286 30735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f781ca57ae9473b858d6d4d6bceeb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.417404 30793 tablet_bootstrap.cc:492] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:41.349401 30796 consensus_queue.cc:260] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.406399 30664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1269e0f1364a69b9a36ce877b0da5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.423627 30796 ts_tablet_manager.cc:1434] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.160s user 0.005s sys 0.002s
I20251025 20:03:41.427544 30796 tablet_bootstrap.cc:492] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:41.407274 30667 tablet_service.cc:1505] Processing CreateTablet for tablet 0d9831caf5d44fc080590d41bd68a083 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251025 20:03:41.431279 30793 tablet_bootstrap.cc:654] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.434620 30592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df036864f934f2d81add520300e9ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.444881 30591 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 31) took 1101 ms. Trace:
I20251025 20:03:41.434737 30667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d9831caf5d44fc080590d41bd68a083. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.445952 30753 raft_consensus.cc:2468] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:41.435963 30590 tablet_service.cc:1505] Processing CreateTablet for tablet 775d2ddde3b44056818240b2a8402e9c (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251025 20:03:41.446810 30740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 50) took 1024 ms. Trace:
I20251025 20:03:41.447242 30570 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:41.447400 30740 rpcz_store.cc:276] 1025 20:03:40.422884 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.209344 (+786460us) service_pool.cc:225] Handling call
1025 20:03:41.253335 (+ 43991us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.309249 (+ 55914us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.355996 (+ 46747us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.446155 (+ 90159us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.446741 (+ 586us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.448202 30590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 775d2ddde3b44056818240b2a8402e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.448412 30807 raft_consensus.cc:2804] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:41.458932 30740 tablet_service.cc:1505] Processing CreateTablet for tablet 4614bdb7f750431fa840cabccf782b4b (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251025 20:03:41.471374 30800 tablet_bootstrap.cc:654] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.445470 30591 rpcz_store.cc:276] 1025 20:03:40.343709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.009196 (+665487us) service_pool.cc:225] Handling call
1025 20:03:41.072650 (+ 63454us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.088481 (+ 15831us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.225624 (+137143us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.419557 (+193933us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.444760 (+ 25203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.488015 30660 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 57) took 1098 ms. Trace:
I20251025 20:03:41.491029 30739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 52) took 1052 ms. Trace:
I20251025 20:03:41.551510 30739 rpcz_store.cc:276] 1025 20:03:40.438412 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.294205 (+855793us) service_pool.cc:225] Handling call
1025 20:03:41.357602 (+ 63397us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.357702 (+ 100us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.378539 (+ 20837us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.490269 (+111730us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.490942 (+ 673us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.552067 30591 tablet_service.cc:1505] Processing CreateTablet for tablet fea0c74c9c0c4a69afe072c45c4cbe99 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251025 20:03:41.553637 30591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea0c74c9c0c4a69afe072c45c4cbe99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.498064 30734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 56) took 1055 ms. Trace:
W20251025 20:03:41.510124 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 62998.4us
W20251025 20:03:41.510763 30489 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 61537.6us
I20251025 20:03:41.512634 30666 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 50) took 1310 ms. Trace:
I20251025 20:03:41.554073 30734 rpcz_store.cc:276] 1025 20:03:40.442562 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.353835 (+911273us) service_pool.cc:225] Handling call
1025 20:03:41.355730 (+ 1895us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.356702 (+ 972us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.400384 (+ 43682us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.497458 (+ 97074us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.497974 (+ 516us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.554546 30666 rpcz_store.cc:276] 1025 20:03:40.243258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.206005 (+962747us) service_pool.cc:225] Handling call
1025 20:03:41.226187 (+ 20182us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.226310 (+ 123us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.377416 (+151106us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.512047 (+134631us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.512555 (+ 508us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.553443 30739 tablet_service.cc:1505] Processing CreateTablet for tablet 0d9831caf5d44fc080590d41bd68a083 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251025 20:03:41.557914 30796 tablet_bootstrap.cc:654] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.515537 30594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 41) took 1168 ms. Trace:
I20251025 20:03:41.558445 30594 rpcz_store.cc:276] 1025 20:03:40.347416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.296404 (+948988us) service_pool.cc:225] Handling call
1025 20:03:41.305222 (+ 8818us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.305332 (+ 110us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.436028 (+130696us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.514962 (+ 78934us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.515469 (+ 507us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.551124 30660 rpcz_store.cc:276] 1025 20:03:40.389631 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.323359 (+933728us) service_pool.cc:225] Handling call
1025 20:03:41.325511 (+ 2152us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.325619 (+ 108us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.416294 (+ 90675us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.487387 (+ 71093us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.487936 (+ 549us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.560757 30594 tablet_service.cc:1505] Processing CreateTablet for tablet 424773f793a4443da2db316842a1f3b5 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251025 20:03:41.512632 30742 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 42) took 1099 ms. Trace:
I20251025 20:03:41.516040 30669 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 52) took 1272 ms. Trace:
I20251025 20:03:41.562088 30669 rpcz_store.cc:276] 1025 20:03:40.244051 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.246648 (+1002597us) service_pool.cc:225] Handling call
1025 20:03:41.248582 (+ 1934us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.248710 (+ 128us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.378913 (+130203us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.515527 (+136614us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.515975 (+ 448us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.517163 30743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 51) took 1094 ms. Trace:
I20251025 20:03:41.520165 30793 tablet_bootstrap.cc:492] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:41.563347 30743 rpcz_store.cc:276] 1025 20:03:40.423283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.291027 (+867744us) service_pool.cc:225] Handling call
1025 20:03:41.390312 (+ 99285us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.390416 (+ 104us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.496044 (+105628us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.516685 (+ 20641us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.517103 (+ 418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.563848 30793 ts_tablet_manager.cc:1403] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.147s user 0.008s sys 0.008s
I20251025 20:03:41.518926 30585 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 37) took 1173 ms. Trace:
I20251025 20:03:41.564358 30585 rpcz_store.cc:276] 1025 20:03:40.346006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.110912 (+764906us) service_pool.cc:225] Handling call
1025 20:03:41.113043 (+ 2131us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.113145 (+ 102us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.376556 (+263411us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.518382 (+141826us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.518862 (+ 480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.527137 30667 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 55) took 1218 ms. Trace:
I20251025 20:03:41.566061 30667 rpcz_store.cc:276] 1025 20:03:40.308918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.288446 (+979528us) service_pool.cc:225] Handling call
1025 20:03:41.434278 (+145832us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.434379 (+ 101us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.518438 (+ 84059us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.526515 (+ 8077us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.527055 (+ 540us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.566443 30585 tablet_service.cc:1505] Processing CreateTablet for tablet e8c971a6d4854180b4d243fa79fbbc11 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251025 20:03:41.567489 30737 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 49) took 1145 ms. Trace:
I20251025 20:03:41.531607 30592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 42) took 1183 ms. Trace:
I20251025 20:03:41.568881 30592 rpcz_store.cc:276] 1025 20:03:40.347734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.345816 (+998082us) service_pool.cc:225] Handling call
1025 20:03:41.434068 (+ 88252us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.434177 (+ 109us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.520469 (+ 86292us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.530994 (+ 10525us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.531531 (+ 537us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.568213 30737 rpcz_store.cc:276] 1025 20:03:40.422432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.155562 (+733130us) service_pool.cc:225] Handling call
1025 20:03:41.157678 (+ 2116us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.157784 (+ 106us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.396880 (+239096us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.566485 (+169605us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.567360 (+ 875us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.570967 30587 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 32) took 1226 ms. Trace:
I20251025 20:03:41.570797 30793 raft_consensus.cc:359] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.570933 30592 tablet_service.cc:1505] Processing CreateTablet for tablet da8a0eaa68044835a4cf94340ca25ac5 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251025 20:03:41.571628 30793 raft_consensus.cc:385] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.572067 30793 raft_consensus.cc:740] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.571458 30587 rpcz_store.cc:276] 1025 20:03:40.344067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.022601 (+678534us) service_pool.cc:225] Handling call
1025 20:03:41.112174 (+ 89573us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b51fd1 00007fd826b2f263 00007fd826addf59 00007fd81ff83bf4 00007fd81ff83b82 00007fd81ff83ab1
1025 20:03:41.112327 (+ 153us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.112415 (+ 88us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.358212 (+245797us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.570374 (+212162us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.570900 (+ 526us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7305600}
I20251025 20:03:41.537096 30741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 46) took 1122 ms. Trace:
I20251025 20:03:41.537096 30590 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 46) took 1101 ms. Trace:
I20251025 20:03:41.537554 30661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 54) took 1244 ms. Trace:
I20251025 20:03:41.533432 30735 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 58) took 1090 ms. Trace:
I20251025 20:03:41.573868 30741 rpcz_store.cc:276] 1025 20:03:40.414309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.025395 (+611086us) service_pool.cc:225] Handling call
1025 20:03:41.098473 (+ 73078us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.098588 (+ 115us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.375584 (+276996us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.536574 (+160990us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.537023 (+ 449us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.574939 30735 rpcz_store.cc:276] 1025 20:03:40.443475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.317279 (+873804us) service_pool.cc:225] Handling call
1025 20:03:41.330075 (+ 12796us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.338140 (+ 8065us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.525476 (+187336us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.532836 (+ 7360us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.533359 (+ 523us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.575295 30793 consensus_queue.cc:260] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.574016 30590 rpcz_store.cc:276] 1025 20:03:40.456705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.434928 (+978223us) service_pool.cc:225] Handling call
1025 20:03:41.447449 (+ 12521us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.447596 (+ 147us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.529043 (+ 81447us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.536572 (+ 7529us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.537021 (+ 449us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.578606 30590 tablet_service.cc:1505] Processing CreateTablet for tablet 2eddbfc136b841098915d8ea2d82956a (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251025 20:03:41.580421 30585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c971a6d4854180b4d243fa79fbbc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.574072 30661 rpcz_store.cc:276] 1025 20:03:40.293234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.261546 (+968312us) service_pool.cc:225] Handling call
1025 20:03:41.263745 (+ 2199us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.263888 (+ 143us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.531409 (+267521us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.537179 (+ 5770us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.537503 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.557229 30734 tablet_service.cc:1505] Processing CreateTablet for tablet 580e75e8dcd949e39aff92ada50b40be (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251025 20:03:41.583267 30592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da8a0eaa68044835a4cf94340ca25ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.584146 30594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 424773f793a4443da2db316842a1f3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.587561 30590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eddbfc136b841098915d8ea2d82956a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.548012 30665 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 56) took 1184 ms. Trace:
I20251025 20:03:41.548347 30664 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 53) took 1296 ms. Trace:
I20251025 20:03:41.588246 30665 rpcz_store.cc:276] 1025 20:03:40.364166 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.319872 (+955706us) service_pool.cc:225] Handling call
1025 20:03:41.321380 (+ 1508us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.321466 (+ 86us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.539702 (+218236us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.547469 (+ 7767us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.547947 (+ 478us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.549882 30668 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 48) took 1425 ms. Trace:
I20251025 20:03:41.551251 30740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4614bdb7f750431fa840cabccf782b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.589933 30668 rpcz_store.cc:276] 1025 20:03:40.124878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.185777 (+1060899us) service_pool.cc:225] Handling call
1025 20:03:41.215035 (+ 29258us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.215259 (+ 224us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.396397 (+181138us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.549400 (+153003us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.549821 (+ 421us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.496897 30588 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 36) took 1151 ms. Trace:
I20251025 20:03:41.562032 30742 rpcz_store.cc:276] 1025 20:03:40.413065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.841217 (+428152us) service_pool.cc:225] Handling call
1025 20:03:40.887874 (+ 46657us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.887978 (+ 104us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.373182 (+485204us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.512041 (+138859us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.512562 (+ 521us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.543140 30593 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 38) took 1196 ms. Trace:
I20251025 20:03:41.588295 30664 rpcz_store.cc:276] 1025 20:03:40.252222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.258888 (+1006666us) service_pool.cc:225] Handling call
1025 20:03:41.346246 (+ 87358us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.346350 (+ 104us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.539512 (+193162us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.547886 (+ 8374us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.548286 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.590946 30588 rpcz_store.cc:276] 1025 20:03:40.345644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.036239 (+690595us) service_pool.cc:225] Handling call
1025 20:03:41.074948 (+ 38709us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b51fd1 00007fd826b2f263 00007fd826addf59 00007fd81ff83bf4 00007fd81ff83b82 00007fd81ff83ab1
1025 20:03:41.110111 (+ 35163us) spinlock_profiling.cc:243] Waited 7.54 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea5077c 00007fd81ea5033b 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b51fd1 00007fd826b2f263 00007fd826addf59 00007fd81ff83bf4 00007fd81ff83b82 00007fd81ff83ab1
1025 20:03:41.111396 (+ 1285us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.111520 (+ 124us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.344362 (+232842us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.496219 (+151857us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.496814 (+ 595us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25051392}
I20251025 20:03:41.590463 30739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d9831caf5d44fc080590d41bd68a083. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.593394 30588 tablet_service.cc:1505] Processing CreateTablet for tablet 8e7a1b97c4aa4812a26b178e32476392 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251025 20:03:41.597669 30588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7a1b97c4aa4812a26b178e32476392. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.598425 30734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580e75e8dcd949e39aff92ada50b40be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.543722 30662 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 49) took 1399 ms. Trace:
I20251025 20:03:41.599198 30662 rpcz_store.cc:276] 1025 20:03:40.144254 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.202978 (+1058724us) service_pool.cc:225] Handling call
1025 20:03:41.205086 (+ 2108us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.205197 (+ 111us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.403084 (+197887us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.543146 (+140062us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.543637 (+ 491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.603780 30793 ts_tablet_manager.cc:1434] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.040s user 0.008s sys 0.000s
I20251025 20:03:41.549093 30807 raft_consensus.cc:697] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:41.608110 30589 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 45) took 1236 ms. Trace:
I20251025 20:03:41.608594 30589 rpcz_store.cc:276] 1025 20:03:40.393336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.355444 (+962108us) service_pool.cc:225] Handling call
1025 20:03:41.363637 (+ 8193us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.363748 (+ 111us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.555599 (+191851us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.607455 (+ 51856us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.608038 (+ 583us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.591413 30593 rpcz_store.cc:276] 1025 20:03:40.346324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.183519 (+837195us) service_pool.cc:225] Handling call
1025 20:03:41.187991 (+ 4472us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.188120 (+ 129us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.307457 (+119337us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.542270 (+234813us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.543071 (+ 801us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.611048 30589 tablet_service.cc:1505] Processing CreateTablet for tablet 3f1269e0f1364a69b9a36ce877b0da5e (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251025 20:03:41.612426 30593 tablet_service.cc:1505] Processing CreateTablet for tablet 43d8dfe478a4430da93c214b20963f9f (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251025 20:03:41.615474 30807 consensus_queue.cc:237] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.616623 30586 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 34) took 1271 ms. Trace:
I20251025 20:03:41.616971 30593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d8dfe478a4430da93c214b20963f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.573856 30587 tablet_service.cc:1505] Processing CreateTablet for tablet 4614bdb7f750431fa840cabccf782b4b (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251025 20:03:41.618710 30587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4614bdb7f750431fa840cabccf782b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.617091 30586 rpcz_store.cc:276] 1025 20:03:40.344735 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.042067 (+697332us) service_pool.cc:225] Handling call
1025 20:03:41.076464 (+ 34397us) spinlock_profiling.cc:243] Waited 3.27 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b51fd1 00007fd826b2f263 00007fd826addf59 00007fd81ff83bf4 00007fd81ff83b82 00007fd81ff83ab1
1025 20:03:41.110398 (+ 33934us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.110565 (+ 167us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.411820 (+301255us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.615948 (+204128us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.616516 (+ 568us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7523584}
I20251025 20:03:41.638677 30589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1269e0f1364a69b9a36ce877b0da5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.643251 30815 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:41.644145 30815 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.648087 30815 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:41.648309 30736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 53) took 1208 ms. Trace:
I20251025 20:03:41.648461 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:41.648885 30739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 55) took 1206 ms. Trace:
I20251025 20:03:41.649297 30739 rpcz_store.cc:276] 1025 20:03:40.442118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.552498 (+1110380us) service_pool.cc:225] Handling call
1025 20:03:41.558120 (+ 5622us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.558233 (+ 113us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.614718 (+ 56485us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.642001 (+ 27283us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.648817 (+ 6816us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12626}
W20251025 20:03:41.650202 30719 leader_election.cc:343] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.651787 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:41.638640 30586 tablet_service.cc:1505] Processing CreateTablet for tablet 0d9831caf5d44fc080590d41bd68a083 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
W20251025 20:03:41.653317 30716 leader_election.cc:343] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.653843 30716 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:41.655158 30793 tablet_bootstrap.cc:492] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:41.656488 30586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d9831caf5d44fc080590d41bd68a083. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.663589 30594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 47) took 1206 ms. Trace:
I20251025 20:03:41.663925 30594 rpcz_store.cc:276] 1025 20:03:40.457145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.559723 (+1102578us) service_pool.cc:225] Handling call
1025 20:03:41.561767 (+ 2044us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.561885 (+ 118us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.636062 (+ 74177us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.663086 (+ 27024us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.663532 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.665812 30594 tablet_service.cc:1505] Processing CreateTablet for tablet 580e75e8dcd949e39aff92ada50b40be (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251025 20:03:41.645917 30501 catalog_manager.cc:5649] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:41.666970 30594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580e75e8dcd949e39aff92ada50b40be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.660061 30815 raft_consensus.cc:2749] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:41.671170 30814 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:41.648697 30736 rpcz_store.cc:276] 1025 20:03:40.439465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.325039 (+885574us) service_pool.cc:225] Handling call
1025 20:03:41.366575 (+ 41536us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.366675 (+ 100us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.565129 (+198454us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.585705 (+ 20576us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.648242 (+ 62537us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.672649 30814 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.677490 30590 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 51) took 1159 ms. Trace:
I20251025 20:03:41.677724 30793 tablet_bootstrap.cc:654] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.677992 30590 rpcz_store.cc:276] 1025 20:03:40.518465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.577594 (+1059129us) service_pool.cc:225] Handling call
1025 20:03:41.579416 (+ 1822us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.579511 (+ 95us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.614513 (+ 35002us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.676866 (+ 62353us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.677410 (+ 544us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.679123 30593 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 56) took 1138 ms. Trace:
I20251025 20:03:41.680359 30590 tablet_service.cc:1505] Processing CreateTablet for tablet 1f781ca57ae9473b858d6d4d6bceeb53 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251025 20:03:41.680501 30593 rpcz_store.cc:276] 1025 20:03:40.540850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.611275 (+1070425us) service_pool.cc:225] Handling call
1025 20:03:41.616433 (+ 5158us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.616537 (+ 104us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.648540 (+ 32003us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.678579 (+ 30039us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.679058 (+ 479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.684196 30738 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 48) took 1269 ms. Trace:
I20251025 20:03:41.687732 30814 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:41.689296 30800 tablet_bootstrap.cc:492] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:41.690305 30585 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 48) took 1232 ms. Trace:
I20251025 20:03:41.690398 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:41.691058 30800 ts_tablet_manager.cc:1403] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.287s user 0.021s sys 0.019s
I20251025 20:03:41.691820 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:41.693661 30645 leader_election.cc:343] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:41.694474 30645 leader_election.cc:343] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.694512 30800 raft_consensus.cc:359] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.694991 30645 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:41.695448 30800 raft_consensus.cc:385] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.695916 30800 raft_consensus.cc:740] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.696187 30814 raft_consensus.cc:2749] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:41.690784 30585 rpcz_store.cc:276] 1025 20:03:40.457452 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.565390 (+1107938us) service_pool.cc:225] Handling call
1025 20:03:41.573187 (+ 7797us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.573288 (+ 101us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.617868 (+ 44580us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.689680 (+ 71812us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.690233 (+ 553us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.697379 30594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 57) took 1156 ms. Trace:
I20251025 20:03:41.697746 30594 rpcz_store.cc:276] 1025 20:03:40.550136 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.664931 (+1114795us) service_pool.cc:225] Handling call
1025 20:03:41.666587 (+ 1656us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.666672 (+ 85us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.688510 (+ 21838us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.696937 (+ 8427us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.697313 (+ 376us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.699139 30796 tablet_bootstrap.cc:492] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:41.699972 30796 ts_tablet_manager.cc:1403] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.273s user 0.015s sys 0.003s
I20251025 20:03:41.700563 30740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 54) took 1252 ms. Trace:
I20251025 20:03:41.700979 30740 rpcz_store.cc:276] 1025 20:03:40.441062 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.457432 (+1016370us) service_pool.cc:225] Handling call
1025 20:03:41.550654 (+ 93222us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.550769 (+ 115us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.628898 (+ 78129us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.693280 (+ 64382us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.693814 (+ 534us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.702860 30796 raft_consensus.cc:359] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.703433 30796 raft_consensus.cc:385] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.703724 30796 raft_consensus.cc:740] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.704408 30796 consensus_queue.cc:260] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.706848 30796 ts_tablet_manager.cc:1434] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20251025 20:03:41.708942 30796 tablet_bootstrap.cc:492] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:41.697674 30800 consensus_queue.cc:260] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.711794 30590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f781ca57ae9473b858d6d4d6bceeb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:03:41.713017 30800 ts_tablet_manager.cc:1434] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20251025 20:03:41.714241 30800 tablet_bootstrap.cc:492] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:41.721987 30588 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 52) took 1203 ms. Trace:
I20251025 20:03:41.722615 30734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 57) took 1279 ms. Trace:
I20251025 20:03:41.724234 30800 tablet_bootstrap.cc:654] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.724256 30588 rpcz_store.cc:276] 1025 20:03:40.519020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.592369 (+1073349us) service_pool.cc:225] Handling call
1025 20:03:41.597134 (+ 4765us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.597240 (+ 106us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.649184 (+ 51944us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.720697 (+ 71513us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.721871 (+ 1174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.724278 30734 rpcz_store.cc:276] 1025 20:03:40.443052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.555285 (+1112233us) service_pool.cc:225] Handling call
1025 20:03:41.583551 (+ 28266us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.583652 (+ 101us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.687903 (+104251us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.722109 (+ 34206us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.722548 (+ 439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.727988 30591 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 49) took 1270 ms. Trace:
I20251025 20:03:41.684669 30738 rpcz_store.cc:276] 1025 20:03:40.421963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.153564 (+731601us) service_pool.cc:225] Handling call
1025 20:03:41.182777 (+ 29213us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.182877 (+ 100us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.502235 (+319358us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.683596 (+181361us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.684120 (+ 524us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.730307 30591 rpcz_store.cc:276] 1025 20:03:40.457752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.550908 (+1093156us) service_pool.cc:225] Handling call
1025 20:03:41.553001 (+ 2093us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.553107 (+ 106us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.636885 (+ 83778us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.723001 (+ 86116us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.727911 (+ 4910us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.732292 30589 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 53) took 1212 ms. Trace:
I20251025 20:03:41.732729 30589 rpcz_store.cc:276] 1025 20:03:40.519492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.609974 (+1090482us) service_pool.cc:225] Handling call
1025 20:03:41.622232 (+ 12258us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.622369 (+ 137us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.709079 (+ 86710us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.724397 (+ 15318us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.732224 (+ 7827us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.733007 30814 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:41.733954 30814 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.737010 30814 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:41.737010 30592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 50) took 1277 ms. Trace:
I20251025 20:03:41.738041 30592 rpcz_store.cc:276] 1025 20:03:40.497579 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.569839 (+1072260us) service_pool.cc:225] Handling call
1025 20:03:41.571869 (+ 2030us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.572104 (+ 235us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.665315 (+ 93211us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.736442 (+ 71127us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.736933 (+ 491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.739683 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:41.743048 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:41.744019 30753 raft_consensus.cc:2468] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
W20251025 20:03:41.746644 30645 leader_election.cc:343] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.747730 30645 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:41.749307 30814 raft_consensus.cc:2804] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:41.749816 30814 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:41.750284 30814 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:41.759562 30793 tablet_bootstrap.cc:492] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:41.760150 30793 ts_tablet_manager.cc:1403] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.007s
I20251025 20:03:41.756898 30590 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 58) took 1206 ms. Trace:
I20251025 20:03:41.762002 30587 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 54) took 1242 ms. Trace:
I20251025 20:03:41.760950 30590 rpcz_store.cc:276] 1025 20:03:40.550545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.679086 (+1128541us) service_pool.cc:225] Handling call
1025 20:03:41.683251 (+ 4165us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.686438 (+ 3187us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.744279 (+ 57841us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.753408 (+ 9129us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.756534 (+ 3126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.763130 30800 tablet_bootstrap.cc:492] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:41.762642 30587 rpcz_store.cc:276] 1025 20:03:40.540053 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.572861 (+1032808us) service_pool.cc:225] Handling call
1025 20:03:41.618223 (+ 45362us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.618334 (+ 111us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.747032 (+128698us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.761217 (+ 14185us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.761845 (+ 628us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.763818 30800 ts_tablet_manager.cc:1403] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.025s
I20251025 20:03:41.764086 30807 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:41.764671 30807 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.763123 30793 raft_consensus.cc:359] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.766587 30793 raft_consensus.cc:385] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.766899 30793 raft_consensus.cc:740] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.766603 30796 tablet_bootstrap.cc:654] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.767709 30800 raft_consensus.cc:359] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.768510 30800 raft_consensus.cc:385] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.768853 30800 raft_consensus.cc:740] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.770532 30800 consensus_queue.cc:260] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.771430 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:41.772023 30793 consensus_queue.cc:260] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.773101 30800 ts_tablet_manager.cc:1434] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20251025 20:03:41.772974 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:41.774305 30800 tablet_bootstrap.cc:492] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
W20251025 20:03:41.775970 30567 leader_election.cc:343] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:41.777199 30570 leader_election.cc:343] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.777657 30570 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:41.779278 30820 raft_consensus.cc:2749] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:41.780040 30807 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:41.782177 30814 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.784391 30793 ts_tablet_manager.cc:1434] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20251025 20:03:41.791318 30793 tablet_bootstrap.cc:492] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:41.792019 30814 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:41.793430 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:41.794203 30753 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:41.799369 30793 tablet_bootstrap.cc:654] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.786330 30800 tablet_bootstrap.cc:654] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.802484 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:41.804400 30796 tablet_bootstrap.cc:492] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:41.805006 30796 ts_tablet_manager.cc:1403] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.096s user 0.013s sys 0.006s
W20251025 20:03:41.805135 30645 leader_election.cc:343] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.806051 30753 raft_consensus.cc:2468] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:41.808635 30645 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:41.810055 30814 raft_consensus.cc:2804] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:41.809644 30796 raft_consensus.cc:359] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.810534 30796 raft_consensus.cc:385] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.810886 30796 raft_consensus.cc:740] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.811820 30796 consensus_queue.cc:260] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.834514 30586 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 55) took 1294 ms. Trace:
I20251025 20:03:41.837224 30586 rpcz_store.cc:276] 1025 20:03:40.540519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.637440 (+1096921us) service_pool.cc:225] Handling call
1025 20:03:41.654633 (+ 17193us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.654954 (+ 321us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.784154 (+129200us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.809072 (+ 24918us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.834413 (+ 25341us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:41.841970 30814 raft_consensus.cc:697] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:41.843468 30793 tablet_bootstrap.cc:492] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:41.844036 30793 ts_tablet_manager.cc:1403] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.053s user 0.017s sys 0.010s
I20251025 20:03:41.841811 30800 tablet_bootstrap.cc:492] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:41.843808 30814 consensus_queue.cc:237] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.845134 30800 ts_tablet_manager.cc:1403] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.071s user 0.025s sys 0.010s
I20251025 20:03:41.846722 30796 ts_tablet_manager.cc:1434] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.041s user 0.016s sys 0.024s
I20251025 20:03:41.848078 30796 tablet_bootstrap.cc:492] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:41.848512 30800 raft_consensus.cc:359] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.849351 30800 raft_consensus.cc:385] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.849684 30800 raft_consensus.cc:740] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.849110 30793 raft_consensus.cc:359] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.850297 30793 raft_consensus.cc:385] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.850533 30800 consensus_queue.cc:260] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.850643 30793 raft_consensus.cc:740] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.851915 30793 consensus_queue.cc:260] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.860121 30796 tablet_bootstrap.cc:654] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.862479 30793 ts_tablet_manager.cc:1434] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.018s user 0.008s sys 0.002s
I20251025 20:03:41.864002 30793 tablet_bootstrap.cc:492] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:41.864231 30800 ts_tablet_manager.cc:1434] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.019s user 0.007s sys 0.001s
I20251025 20:03:41.866515 30807 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:41.867090 30807 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:41.871119 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:41.872558 30800 tablet_bootstrap.cc:492] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
W20251025 20:03:41.872666 30567 leader_election.cc:343] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.873448 30807 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:41.873450 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:41.874675 30753 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:41.876251 30570 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:41.876888 30793 tablet_bootstrap.cc:654] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.877770 30807 raft_consensus.cc:2804] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:41.878422 30807 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:41.878917 30807 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:41.884271 30502 catalog_manager.cc:5649] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:41.894323 30814 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:41.893558 30807 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:41.900628 30814 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.901733 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:41.902061 30800 tablet_bootstrap.cc:654] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.902375 30753 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:41.903378 30796 tablet_bootstrap.cc:492] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:41.903891 30796 ts_tablet_manager.cc:1403] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.056s user 0.016s sys 0.016s
I20251025 20:03:41.907929 30796 raft_consensus.cc:359] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.908721 30796 raft_consensus.cc:385] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.909011 30796 raft_consensus.cc:740] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.910159 30807 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:41.911082 30796 consensus_queue.cc:260] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.911499 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:03:41.913023 30567 leader_election.cc:343] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.914168 30796 ts_tablet_manager.cc:1434] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20251025 20:03:41.916085 30796 tablet_bootstrap.cc:492] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:41.916863 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:41.918373 30645 leader_election.cc:343] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.920730 30753 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:41.922305 30570 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:41.923190 30807 raft_consensus.cc:2804] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:41.923663 30807 raft_consensus.cc:697] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:41.924523 30807 consensus_queue.cc:237] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:41.925808 30814 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:41.927389 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:41.929508 30645 leader_election.cc:343] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.931643 30645 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:41.933252 30796 tablet_bootstrap.cc:654] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.933547 30814 raft_consensus.cc:2749] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:41.935989 30793 tablet_bootstrap.cc:492] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:41.936637 30793 ts_tablet_manager.cc:1403] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.073s user 0.024s sys 0.036s
I20251025 20:03:41.943583 30815 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:41.944059 30815 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.944950 30501 catalog_manager.cc:5649] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:41.946206 30815 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:41.946601 30796 tablet_bootstrap.cc:492] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:41.947187 30796 ts_tablet_manager.cc:1403] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.007s
I20251025 20:03:41.947896 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:41.948396 30679 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:41.948482 30800 tablet_bootstrap.cc:492] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:41.941183 30793 raft_consensus.cc:359] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.949167 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:41.949587 30793 raft_consensus.cc:385] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.950364 30716 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:03:41.950968 30719 leader_election.cc:343] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.950551 30796 raft_consensus.cc:359] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.951301 30815 raft_consensus.cc:2804] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:41.951406 30796 raft_consensus.cc:385] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.951752 30815 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:41.951774 30796 raft_consensus.cc:740] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.952244 30815 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:41.952659 30796 consensus_queue.cc:260] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.955513 30796 ts_tablet_manager.cc:1434] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20251025 20:03:41.958096 30796 tablet_bootstrap.cc:492] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:41.949949 30793 raft_consensus.cc:740] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.962730 30793 consensus_queue.cc:260] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.949121 30800 ts_tablet_manager.cc:1403] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.013s
I20251025 20:03:41.966663 30796 tablet_bootstrap.cc:654] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.970320 30793 ts_tablet_manager.cc:1434] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.033s user 0.008s sys 0.002s
I20251025 20:03:41.971541 30793 tablet_bootstrap.cc:492] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:41.974552 30815 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.968618 30800 raft_consensus.cc:359] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.979044 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:41.979115 30800 raft_consensus.cc:385] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.979715 30800 raft_consensus.cc:740] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.980893 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:41.981622 30679 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:41.980927 30800 consensus_queue.cc:260] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.984725 30800 ts_tablet_manager.cc:1434] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.019s user 0.001s sys 0.006s
I20251025 20:03:41.985448 30815 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:41.986780 30719 leader_election.cc:343] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:41.989377 30679 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:41.990640 30796 tablet_bootstrap.cc:492] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:41.990705 30716 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:41.991261 30796 ts_tablet_manager.cc:1403] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.006s
I20251025 20:03:41.992368 30815 raft_consensus.cc:2804] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:41.993734 30815 raft_consensus.cc:697] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:41.994679 30815 consensus_queue.cc:237] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.995647 30793 tablet_bootstrap.cc:654] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:41.991271 30800 tablet_bootstrap.cc:492] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:41.995260 30796 raft_consensus.cc:359] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:41.998621 30796 raft_consensus.cc:385] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:41.998981 30842 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:41.998996 30796 raft_consensus.cc:740] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:41.999595 30842 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.000010 30796 consensus_queue.cc:260] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.001863 30842 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.002522 30796 ts_tablet_manager.cc:1434] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20251025 20:03:42.003810 30796 tablet_bootstrap.cc:492] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:42.004899 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.005609 30603 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:42.005846 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.006623 30679 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:42.007061 30719 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:42.008038 30842 raft_consensus.cc:2804] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:42.008400 30842 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:42.008810 30842 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.012751 30796 tablet_bootstrap.cc:654] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.014268 30800 tablet_bootstrap.cc:654] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.014997 30793 tablet_bootstrap.cc:492] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:42.015717 30793 ts_tablet_manager.cc:1403] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.004s
I20251025 20:03:42.021651 30814 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.030797 30814 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.032684 30814 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.034767 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.034801 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:42.035647 30603 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:42.035794 30753 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:42.037196 30645 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:42.038743 30814 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:42.039170 30814 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:42.039618 30814 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.040786 30793 raft_consensus.cc:359] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.041527 30793 raft_consensus.cc:385] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.041812 30793 raft_consensus.cc:740] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.043314 30796 tablet_bootstrap.cc:492] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:42.043857 30796 ts_tablet_manager.cc:1403] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.040s user 0.021s sys 0.003s
I20251025 20:03:42.043519 30793 consensus_queue.cc:260] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.044374 30500 catalog_manager.cc:5649] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:42.046128 30793 ts_tablet_manager.cc:1434] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20251025 20:03:42.047202 30793 tablet_bootstrap.cc:492] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:42.053119 30842 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.056609 30796 raft_consensus.cc:359] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.057245 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:42.057466 30796 raft_consensus.cc:385] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.057865 30603 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.057909 30796 raft_consensus.cc:740] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.058998 30796 consensus_queue.cc:260] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.061298 30796 ts_tablet_manager.cc:1434] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20251025 20:03:42.061659 30793 tablet_bootstrap.cc:654] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.062386 30796 tablet_bootstrap.cc:492] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:42.069289 30796 tablet_bootstrap.cc:654] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.076152 30814 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.078572 30842 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.080248 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:42.081136 30679 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.082218 30793 tablet_bootstrap.cc:492] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:42.082523 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:42.082757 30793 ts_tablet_manager.cc:1403] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.036s user 0.023s sys 0.003s
I20251025 20:03:42.083194 30602 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.083352 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:42.084232 30753 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.086838 30793 raft_consensus.cc:359] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.087508 30793 raft_consensus.cc:385] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.087795 30793 raft_consensus.cc:740] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.088651 30793 consensus_queue.cc:260] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.090785 30793 ts_tablet_manager.cc:1434] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251025 20:03:42.091773 30793 tablet_bootstrap.cc:492] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:42.092813 30753 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:42.093518 30602 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:42.094172 30645 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:42.076200 30603 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:42.096256 30834 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:42.096961 30834 raft_consensus.cc:697] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:42.098701 30719 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:42.099733 30842 raft_consensus.cc:2804] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:42.100231 30842 raft_consensus.cc:697] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:42.101294 30793 tablet_bootstrap.cc:654] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.101082 30842 consensus_queue.cc:237] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.102272 30834 consensus_queue.cc:237] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.094203 30679 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:42.109330 30835 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.105618 30814 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.110050 30835 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.119138 30835 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.120249 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.122186 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:42.123359 30753 raft_consensus.cc:2393] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:42.124504 30645 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:42.126124 30814 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.131104 30800 tablet_bootstrap.cc:492] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:42.131703 30800 ts_tablet_manager.cc:1403] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.142s user 0.013s sys 0.021s
I20251025 20:03:42.134137 30796 tablet_bootstrap.cc:492] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:42.134680 30796 ts_tablet_manager.cc:1403] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.025s
I20251025 20:03:42.135862 30800 raft_consensus.cc:359] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.137059 30800 raft_consensus.cc:385] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.137442 30800 raft_consensus.cc:740] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.137974 30796 raft_consensus.cc:359] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:42.138778 30814 raft_consensus.cc:2749] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:42.138638 30800 consensus_queue.cc:260] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.138687 30796 raft_consensus.cc:385] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.142426 30796 raft_consensus.cc:740] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.143388 30796 consensus_queue.cc:260] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:42.146322 30815 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.147032 30800 ts_tablet_manager.cc:1434] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20251025 20:03:42.146901 30815 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.149183 30815 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.149691 30800 tablet_bootstrap.cc:492] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:42.154048 30796 ts_tablet_manager.cc:1434] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20251025 20:03:42.154459 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.155483 30796 tablet_bootstrap.cc:492] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:42.155434 30679 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:42.158242 30716 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:42.160899 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.162698 30815 raft_consensus.cc:2804] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:42.165120 30500 catalog_manager.cc:5649] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:42.161819 30603 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:42.163988 30815 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:42.170845 30815 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.177837 30807 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.179512 30807 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.179831 30793 tablet_bootstrap.cc:492] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:42.182518 30793 ts_tablet_manager.cc:1403] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.091s user 0.032s sys 0.013s
I20251025 20:03:42.181684 30807 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.183370 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.183591 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:42.184407 30753 raft_consensus.cc:2393] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:42.186136 30800 tablet_bootstrap.cc:654] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.186254 30570 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:42.182628 30842 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.190272 30842 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.192210 30842 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.191967 30815 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.192804 30807 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.193552 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:42.195081 30719 leader_election.cc:343] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.195536 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:42.196303 30603 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.198421 30815 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.197679 30793 raft_consensus.cc:359] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.199483 30793 raft_consensus.cc:385] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.199807 30796 tablet_bootstrap.cc:654] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.199806 30793 raft_consensus.cc:740] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.199162 30820 consensus_queue.cc:1048] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:42.201426 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.202126 30679 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:42.203449 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:42.202999 30500 catalog_manager.cc:5649] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:42.204236 30716 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:42.205370 30815 raft_consensus.cc:2804] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:42.206154 30679 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.206676 30815 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:42.207080 30815 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.215708 30815 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.200686 30793 consensus_queue.cc:260] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.218654 30793 ts_tablet_manager.cc:1434] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.036s user 0.007s sys 0.001s
I20251025 20:03:42.219125 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:03:42.220672 30719 leader_election.cc:343] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.205119 30807 raft_consensus.cc:2749] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:42.221531 30679 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:42.223354 30603 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:42.229339 30807 consensus_queue.cc:1048] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251025 20:03:42.235898 30719 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:42.237207 30842 raft_consensus.cc:2804] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:42.223142 30793 tablet_bootstrap.cc:492] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:42.253515 30800 tablet_bootstrap.cc:492] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:42.254379 30800 ts_tablet_manager.cc:1403] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.105s user 0.022s sys 0.000s
I20251025 20:03:42.231648 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:42.256383 30677 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.230610 30815 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.280284 30793 tablet_bootstrap.cc:654] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.282001 30800 raft_consensus.cc:359] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.283018 30800 raft_consensus.cc:385] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.283367 30800 raft_consensus.cc:740] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.269881 30842 raft_consensus.cc:697] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:42.284567 30815 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.284967 30842 consensus_queue.cc:237] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.285251 30815 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.288477 30820 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.289090 30820 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.289853 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.290541 30603 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:42.291967 30820 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.291848 30719 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:42.288504 30815 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.293597 30815 raft_consensus.cc:2804] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:42.293813 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.293354 30800 consensus_queue.cc:260] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.294850 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:42.295017 30815 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:03:42.296917 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:42.297142 30676 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:42.293351 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.297394 30815 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.299928 30678 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:42.303364 30800 ts_tablet_manager.cc:1434] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.049s user 0.010s sys 0.000s
I20251025 20:03:42.300568 30834 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.311498 30800 tablet_bootstrap.cc:492] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:42.323092 30796 tablet_bootstrap.cc:492] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:42.316084 30567 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:42.324034 30796 ts_tablet_manager.cc:1403] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.169s user 0.042s sys 0.045s
I20251025 20:03:42.324254 30807 raft_consensus.cc:2804] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:42.324663 30807 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:42.325595 30807 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.328974 30815 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.315487 30834 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.330360 30752 raft_consensus.cc:2393] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:42.331866 30834 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.333678 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.337713 30815 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.338150 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:42.333956 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:42.338687 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:42.339140 30751 raft_consensus.cc:2468] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:42.339497 30676 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.333722 30677 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:42.343585 30645 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:42.344642 30834 raft_consensus.cc:2804] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:42.344724 30716 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:42.345055 30834 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:42.345616 30834 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:42.345129 30645 leader_election.cc:343] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.349475 30796 raft_consensus.cc:359] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.350230 30796 raft_consensus.cc:385] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.357599 30796 raft_consensus.cc:740] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.358453 30796 consensus_queue.cc:260] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.361258 30814 consensus_queue.cc:1048] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:42.363530 30676 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:42.369050 30796 ts_tablet_manager.cc:1434] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.045s user 0.002s sys 0.004s
I20251025 20:03:42.356228 30815 raft_consensus.cc:2804] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:42.370887 30796 tablet_bootstrap.cc:492] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:42.373601 30820 consensus_queue.cc:1048] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251025 20:03:42.376005 30793 tablet_bootstrap.cc:492] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:42.376528 30793 ts_tablet_manager.cc:1403] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.154s user 0.014s sys 0.001s
I20251025 20:03:42.378018 30716 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:42.381057 30859 raft_consensus.cc:2804] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:42.381572 30859 raft_consensus.cc:697] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:42.382730 30859 consensus_queue.cc:237] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.373368 30815 raft_consensus.cc:697] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:42.380511 30793 raft_consensus.cc:359] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.386170 30793 raft_consensus.cc:385] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.386478 30793 raft_consensus.cc:740] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.387038 30793 consensus_queue.cc:260] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.391651 30856 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.392318 30793 ts_tablet_manager.cc:1434] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20251025 20:03:42.392228 30856 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.393361 30793 tablet_bootstrap.cc:492] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:42.392978 30815 consensus_queue.cc:237] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.402485 30793 tablet_bootstrap.cc:654] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.392567 30834 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.406788 30677 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.416132 30834 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.416827 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.415760 30501 catalog_manager.cc:5649] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:42.403614 30856 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.437059 30800 tablet_bootstrap.cc:654] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.417711 30834 consensus_queue.cc:1048] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20251025 20:03:42.429912 30719 leader_election.cc:343] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.434520 30807 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.471324 30796 tablet_bootstrap.cc:654] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.469684 30807 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.478883 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:42.481535 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:42.482403 30679 raft_consensus.cc:2393] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:42.483531 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:42.473615 30856 consensus_queue.cc:1048] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20251025 20:03:42.487054 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:42.478852 30602 raft_consensus.cc:2393] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:42.490172 30567 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:42.492157 30719 consensus_peers.cc:597] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:42.487740 30752 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.496105 30857 raft_consensus.cc:2749] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:03:42.496352 30807 consensus_queue.cc:1048] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:42.497133 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:42.499461 30716 leader_election.cc:343] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.500002 30716 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:42.501101 30856 raft_consensus.cc:2749] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:42.525429 30796 tablet_bootstrap.cc:492] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:42.526394 30796 ts_tablet_manager.cc:1403] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.156s user 0.011s sys 0.009s
W20251025 20:03:42.514410 30645 leader_election.cc:343] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.532897 30793 tablet_bootstrap.cc:492] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:42.533609 30793 ts_tablet_manager.cc:1403] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.140s user 0.013s sys 0.004s
I20251025 20:03:42.538375 30752 raft_consensus.cc:2468] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:42.542520 30796 raft_consensus.cc:359] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.543387 30796 raft_consensus.cc:385] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.543726 30796 raft_consensus.cc:740] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.546459 30796 consensus_queue.cc:260] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.549019 30645 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:42.550077 30834 raft_consensus.cc:2804] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:42.550613 30834 raft_consensus.cc:697] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:42.551647 30834 consensus_queue.cc:237] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.554071 30815 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.554813 30815 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.560112 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.562052 30815 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.566231 30793 raft_consensus.cc:359] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.582082 30793 raft_consensus.cc:385] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.549067 30796 ts_tablet_manager.cc:1434] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20251025 20:03:42.572242 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.582438 30793 raft_consensus.cc:740] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.584290 30793 consensus_queue.cc:260] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:42.570322 30719 leader_election.cc:343] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.585842 30796 tablet_bootstrap.cc:492] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
W20251025 20:03:42.586279 30716 leader_election.cc:343] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.586705 30716 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:42.587905 30815 raft_consensus.cc:2749] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:42.591243 30793 ts_tablet_manager.cc:1434] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.057s user 0.005s sys 0.003s
I20251025 20:03:42.593572 30793 tablet_bootstrap.cc:492] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:42.599069 30502 catalog_manager.cc:5649] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:42.598889 30500 catalog_manager.cc:5649] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:42.602851 30800 tablet_bootstrap.cc:492] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:42.603375 30800 ts_tablet_manager.cc:1403] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.292s user 0.050s sys 0.013s
I20251025 20:03:42.602360 30500 catalog_manager.cc:5649] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:42.606498 30800 raft_consensus.cc:359] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.607206 30800 raft_consensus.cc:385] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.607539 30800 raft_consensus.cc:740] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.608246 30800 consensus_queue.cc:260] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.608937 30796 tablet_bootstrap.cc:654] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.610486 30800 ts_tablet_manager.cc:1434] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251025 20:03:42.611524 30800 tablet_bootstrap.cc:492] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:42.614549 30793 tablet_bootstrap.cc:654] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.619894 30796 tablet_bootstrap.cc:492] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:42.620441 30796 ts_tablet_manager.cc:1403] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.005s
I20251025 20:03:42.624162 30796 raft_consensus.cc:359] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.624900 30796 raft_consensus.cc:385] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.625197 30796 raft_consensus.cc:740] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.625950 30796 consensus_queue.cc:260] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.628350 30796 ts_tablet_manager.cc:1434] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20251025 20:03:42.628798 30793 tablet_bootstrap.cc:492] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:42.629290 30793 ts_tablet_manager.cc:1403] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.010s
I20251025 20:03:42.629412 30796 tablet_bootstrap.cc:492] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:42.631049 30800 tablet_bootstrap.cc:654] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.632354 30793 raft_consensus.cc:359] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.633071 30793 raft_consensus.cc:385] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.633294 30793 raft_consensus.cc:740] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.633865 30793 consensus_queue.cc:260] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.637260 30793 ts_tablet_manager.cc:1434] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20251025 20:03:42.638257 30793 tablet_bootstrap.cc:492] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:42.640925 30834 consensus_queue.cc:1048] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:42.641328 30796 tablet_bootstrap.cc:654] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.647212 30815 consensus_queue.cc:1048] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:42.650985 30815 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.651468 30815 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.652976 30800 tablet_bootstrap.cc:492] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:42.653546 30800 ts_tablet_manager.cc:1403] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.003s
I20251025 20:03:42.657770 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.658603 30800 raft_consensus.cc:359] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.659554 30800 raft_consensus.cc:385] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.659924 30800 raft_consensus.cc:740] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.660673 30834 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.661237 30834 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.663250 30793 tablet_bootstrap.cc:654] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.663343 30834 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.653920 30815 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.664603 30800 consensus_queue.cc:260] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.669061 30800 ts_tablet_manager.cc:1434] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20251025 20:03:42.670217 30800 tablet_bootstrap.cc:492] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:42.671839 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:42.665964 30719 leader_election.cc:343] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.674345 30750 raft_consensus.cc:2468] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:42.674597 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.675364 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.675632 30645 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:03:42.676059 30716 leader_election.cc:343] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.673125 30835 consensus_queue.cc:1048] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:42.676515 30716 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:42.679988 30868 raft_consensus.cc:2804] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:42.680531 30859 raft_consensus.cc:2749] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:42.680964 30815 consensus_queue.cc:1048] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:42.682688 30796 tablet_bootstrap.cc:492] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:42.683174 30796 ts_tablet_manager.cc:1403] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.054s user 0.016s sys 0.010s
I20251025 20:03:42.683874 30868 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:42.687203 30868 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.687366 30815 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:03:42.686125 30645 leader_election.cc:343] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.688045 30815 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.691530 30815 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:42.694936 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.697005 30796 raft_consensus.cc:359] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.698060 30796 raft_consensus.cc:385] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.698357 30796 raft_consensus.cc:740] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.699371 30814 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.699146 30796 consensus_queue.cc:260] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.699888 30814 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.701954 30796 ts_tablet_manager.cc:1434] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20251025 20:03:42.703859 30814 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.706300 30796 tablet_bootstrap.cc:492] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:42.710430 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:42.710182 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:42.712007 30645 leader_election.cc:343] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.712069 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:42.708096 30719 leader_election.cc:343] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:42.714493 30645 leader_election.cc:343] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.718873 30645 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:42.722122 30814 raft_consensus.cc:2749] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 20:03:42.722947 30716 leader_election.cc:343] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.723428 30716 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:42.726754 30867 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.727278 30867 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.728766 30859 raft_consensus.cc:2749] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:42.730053 30867 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.734517 30859 consensus_queue.cc:1048] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:42.735673 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:42.736296 30814 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.737142 30814 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.740037 30814 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.738175 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.746541 30677 raft_consensus.cc:2393] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:42.747200 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:42.751549 30645 leader_election.cc:343] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.753494 30567 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:42.754024 30796 tablet_bootstrap.cc:654] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.758862 30807 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.759690 30867 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.760001 30807 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.763988 30793 tablet_bootstrap.cc:492] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:42.764009 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:42.765878 30750 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:42.764518 30793 ts_tablet_manager.cc:1403] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.126s user 0.027s sys 0.014s
I20251025 20:03:42.766944 30645 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:42.768014 30807 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.768467 30835 raft_consensus.cc:2804] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:42.768950 30835 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:42.769277 30835 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.769399 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.768277 30842 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.770694 30842 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.773926 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:42.775650 30719 leader_election.cc:343] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.775985 30842 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:42.778736 30567 leader_election.cc:343] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.779174 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:42.783053 30800 tablet_bootstrap.cc:654] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.783398 30815 consensus_queue.cc:1048] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:42.787333 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:42.788765 30570 leader_election.cc:343] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.789391 30570 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:42.790846 30793 raft_consensus.cc:359] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.791383 30793 raft_consensus.cc:385] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.791584 30793 raft_consensus.cc:740] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.792624 30807 raft_consensus.cc:2749] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:42.794646 30868 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.796983 30868 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.798316 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:42.798247 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:42.799669 30750 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.800423 30793 consensus_queue.cc:260] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:42.800096 30645 leader_election.cc:343] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.803906 30859 consensus_queue.cc:1048] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251025 20:03:42.809693 30793 ts_tablet_manager.cc:1434] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.043s user 0.005s sys 0.002s
I20251025 20:03:42.810879 30793 tablet_bootstrap.cc:492] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:42.811019 30867 raft_consensus.cc:2749] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:42.842640 30859 consensus_queue.cc:1048] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:42.841850 30835 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.846838 30677 raft_consensus.cc:2393] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:42.847484 30835 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.850989 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:42.851686 30753 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.855557 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:42.857621 30716 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:03:42.858111 30645 leader_election.cc:343] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.872788 30815 consensus_queue.cc:1048] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251025 20:03:42.876874 30750 raft_consensus.cc:2468] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:42.883633 30842 consensus_queue.cc:1048] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251025 20:03:42.882560 30645 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:42.899843 30868 raft_consensus.cc:2804] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:42.900420 30868 raft_consensus.cc:697] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:42.901523 30793 tablet_bootstrap.cc:654] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.901448 30868 consensus_queue.cc:237] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.897646 30753 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:42.914542 30800 tablet_bootstrap.cc:492] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:42.917472 30800 ts_tablet_manager.cc:1403] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.248s user 0.023s sys 0.012s
I20251025 20:03:42.917344 30859 raft_consensus.cc:2749] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:42.922781 30604 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:42.924597 30645 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:42.925980 30814 raft_consensus.cc:2804] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:42.926493 30814 raft_consensus.cc:697] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:42.929311 30814 consensus_queue.cc:237] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.935057 30800 raft_consensus.cc:359] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.935642 30800 raft_consensus.cc:385] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.935916 30800 raft_consensus.cc:740] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.936678 30800 consensus_queue.cc:260] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.951088 30800 ts_tablet_manager.cc:1434] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20251025 20:03:42.951743 30796 tablet_bootstrap.cc:492] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:42.952312 30796 ts_tablet_manager.cc:1403] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.249s user 0.026s sys 0.029s
I20251025 20:03:42.952787 30800 tablet_bootstrap.cc:492] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:42.962265 30796 raft_consensus.cc:359] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.964228 30796 raft_consensus.cc:385] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:42.964619 30796 raft_consensus.cc:740] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:42.965478 30796 consensus_queue.cc:260] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.984004 30835 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:42.984792 30835 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:42.989993 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:42.991775 30645 leader_election.cc:343] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:42.993000 30835 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:42.994967 30800 tablet_bootstrap.cc:654] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:42.996275 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:42.997054 30750 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:43.002777 30645 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:43.006762 30796 ts_tablet_manager.cc:1434] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.054s user 0.007s sys 0.003s
I20251025 20:03:43.008189 30796 tablet_bootstrap.cc:492] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:43.009845 30859 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.010543 30868 raft_consensus.cc:2804] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:43.010396 30859 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.020377 30859 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:43.021711 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:43.022152 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:43.022496 30603 raft_consensus.cc:2468] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:43.016773 30868 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:43.024363 30719 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:43.024716 30868 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:43.025475 30716 leader_election.cc:343] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.030936 30859 raft_consensus.cc:2804] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:43.031584 30859 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:43.032044 30859 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.037508 30500 catalog_manager.cc:5649] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:43.055644 30835 consensus_queue.cc:1048] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:43.067322 30859 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.071213 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:43.071954 30603 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.080024 30796 tablet_bootstrap.cc:654] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.079707 30868 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:43.080714 30645 consensus_peers.cc:597] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:43.071481 30793 tablet_bootstrap.cc:492] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:43.082020 30793 ts_tablet_manager.cc:1403] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.271s user 0.035s sys 0.042s
I20251025 20:03:43.083007 30807 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:43.083527 30807 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.084616 30859 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:43.084894 30793 raft_consensus.cc:359] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.085644 30793 raft_consensus.cc:385] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.085980 30793 raft_consensus.cc:740] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.086673 30793 consensus_queue.cc:260] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.088986 30793 ts_tablet_manager.cc:1434] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251025 20:03:43.088325 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:43.090024 30793 tablet_bootstrap.cc:492] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:43.091406 30868 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.091552 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:43.089797 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:43.093350 30750 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:43.093578 30716 leader_election.cc:343] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.094661 30603 raft_consensus.cc:2468] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:43.096195 30719 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
W20251025 20:03:43.098265 30645 leader_election.cc:343] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.099086 30859 raft_consensus.cc:2804] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:43.099613 30859 raft_consensus.cc:697] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:43.101037 30859 consensus_queue.cc:237] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.104151 30807 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.103928 30501 catalog_manager.cc:5649] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:43.107043 30800 tablet_bootstrap.cc:492] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:43.109413 30800 ts_tablet_manager.cc:1403] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.157s user 0.014s sys 0.009s
I20251025 20:03:43.112464 30800 raft_consensus.cc:359] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.131026 30800 raft_consensus.cc:385] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.131513 30800 raft_consensus.cc:740] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.132480 30800 consensus_queue.cc:260] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.115528 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:43.135082 30796 tablet_bootstrap.cc:492] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:43.135644 30796 ts_tablet_manager.cc:1403] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.128s user 0.030s sys 0.022s
I20251025 20:03:43.136927 30793 tablet_bootstrap.cc:654] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.138491 30796 raft_consensus.cc:359] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.117214 30750 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:43.115643 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:43.141422 30570 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:43.143265 30807 raft_consensus.cc:2749] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:43.144167 30807 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.144661 30645 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:43.144752 30807 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.146355 30899 raft_consensus.cc:2804] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:43.146854 30899 raft_consensus.cc:697] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:43.147626 30899 consensus_queue.cc:237] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.139269 30796 raft_consensus.cc:385] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.138728 30800 ts_tablet_manager.cc:1434] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
I20251025 20:03:43.151469 30796 raft_consensus.cc:740] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.152896 30796 consensus_queue.cc:260] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.157162 30796 ts_tablet_manager.cc:1434] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.021s user 0.009s sys 0.001s
I20251025 20:03:43.159144 30796 tablet_bootstrap.cc:492] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:43.165304 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:43.167596 30567 leader_election.cc:343] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.169483 30796 tablet_bootstrap.cc:654] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.170614 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:43.171648 30753 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:43.173775 30570 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:43.175012 30902 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:43.175160 30807 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.175640 30902 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:43.176170 30902 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.181973 30501 catalog_manager.cc:5649] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:43.183871 30800 tablet_bootstrap.cc:492] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:43.193747 30800 tablet_bootstrap.cc:654] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.200412 30793 tablet_bootstrap.cc:492] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:43.200886 30793 ts_tablet_manager.cc:1403] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.111s user 0.023s sys 0.023s
I20251025 20:03:43.204679 30793 raft_consensus.cc:359] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.206200 30793 raft_consensus.cc:385] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.206585 30793 raft_consensus.cc:740] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.207362 30793 consensus_queue.cc:260] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.201749 30902 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.211439 30793 ts_tablet_manager.cc:1434] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20251025 20:03:43.211529 30902 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.213374 30793 tablet_bootstrap.cc:492] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:43.214984 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:43.215714 30753 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.215936 30501 catalog_manager.cc:5649] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:43.222235 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:03:43.223646 30567 leader_election.cc:343] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.223750 30793 tablet_bootstrap.cc:654] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.241694 30753 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:43.243288 30570 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:43.244674 30902 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:43.246127 30902 raft_consensus.cc:697] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:43.246975 30902 consensus_queue.cc:237] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.244982 30800 tablet_bootstrap.cc:492] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:43.249800 30800 ts_tablet_manager.cc:1403] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.005s
I20251025 20:03:43.254500 30800 raft_consensus.cc:359] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.255530 30800 raft_consensus.cc:385] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.255896 30800 raft_consensus.cc:740] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.256450 30899 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.256964 30899 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.258828 30796 tablet_bootstrap.cc:492] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:43.259505 30796 ts_tablet_manager.cc:1403] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.101s user 0.040s sys 0.048s
I20251025 20:03:43.259531 30899 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.261344 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:43.261263 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:43.263285 30796 raft_consensus.cc:359] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:43.264146 30645 leader_election.cc:343] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.264279 30796 raft_consensus.cc:385] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.264698 30796 raft_consensus.cc:740] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.265237 30753 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:43.265754 30800 consensus_queue.cc:260] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.266394 30645 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:43.266490 30500 catalog_manager.cc:5649] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:43.267849 30899 raft_consensus.cc:2804] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:43.268232 30899 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:43.268569 30899 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.269062 30800 ts_tablet_manager.cc:1434] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20251025 20:03:43.270498 30800 tablet_bootstrap.cc:492] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:43.274246 30796 consensus_queue.cc:260] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.277246 30796 ts_tablet_manager.cc:1434] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.017s user 0.006s sys 0.004s
I20251025 20:03:43.278780 30796 tablet_bootstrap.cc:492] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:43.281421 30800 tablet_bootstrap.cc:654] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.279799 30899 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.286726 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:43.287885 30793 tablet_bootstrap.cc:492] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:43.288625 30793 ts_tablet_manager.cc:1403] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.076s user 0.035s sys 0.027s
W20251025 20:03:43.289987 30645 leader_election.cc:343] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.292243 30899 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.293305 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:43.294059 30753 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.298432 30793 raft_consensus.cc:359] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.299680 30793 raft_consensus.cc:385] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.300129 30793 raft_consensus.cc:740] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.300679 30859 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.301182 30859 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.317067 30899 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.317278 30793 consensus_queue.cc:260] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.317689 30899 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.319962 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:43.321291 30793 ts_tablet_manager.cc:1434] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.032s user 0.017s sys 0.009s
W20251025 20:03:43.321513 30719 leader_election.cc:343] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.323076 30793 tablet_bootstrap.cc:492] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:43.324316 30800 tablet_bootstrap.cc:492] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:43.324821 30800 ts_tablet_manager.cc:1403] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.055s user 0.014s sys 0.009s
I20251025 20:03:43.322988 30859 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:43.326085 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:43.326709 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:43.327356 30676 raft_consensus.cc:2468] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
W20251025 20:03:43.328380 30645 leader_election.cc:343] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.328989 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:43.329438 30716 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:43.329687 30899 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.329461 30835 consensus_queue.cc:1048] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20251025 20:03:43.331130 30645 leader_election.cc:343] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.331630 30645 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:43.332365 30859 raft_consensus.cc:2804] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:43.332739 30859 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:43.333112 30835 raft_consensus.cc:2749] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:43.332942 30800 raft_consensus.cc:359] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.333861 30800 raft_consensus.cc:385] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.334167 30800 raft_consensus.cc:740] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.334684 30800 consensus_queue.cc:260] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.338797 30800 ts_tablet_manager.cc:1434] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20251025 20:03:43.333122 30859 raft_consensus.cc:3060] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.340741 30800 tablet_bootstrap.cc:492] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:43.352396 30796 tablet_bootstrap.cc:654] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:43.354399 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:43.354581 30753 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:43.356555 30645 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:43.358305 30835 raft_consensus.cc:2804] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:43.358726 30793 tablet_bootstrap.cc:654] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.358848 30835 raft_consensus.cc:697] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:43.359928 30835 consensus_queue.cc:237] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.355614 30859 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.370224 30859 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:43.371556 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:43.372951 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:43.373564 30676 raft_consensus.cc:3060] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.373929 30604 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:43.373924 30719 leader_election.cc:343] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.381917 30800 tablet_bootstrap.cc:654] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.387562 30868 consensus_queue.cc:1048] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:43.390424 30793 tablet_bootstrap.cc:492] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:43.391070 30793 ts_tablet_manager.cc:1403] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.068s user 0.012s sys 0.013s
I20251025 20:03:43.391613 30796 tablet_bootstrap.cc:492] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:43.392128 30796 ts_tablet_manager.cc:1403] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.114s user 0.013s sys 0.010s
I20251025 20:03:43.394011 30793 raft_consensus.cc:359] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.394729 30793 raft_consensus.cc:385] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.394693 30796 raft_consensus.cc:359] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.395272 30793 raft_consensus.cc:740] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.395538 30796 raft_consensus.cc:385] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.395927 30796 raft_consensus.cc:740] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.396561 30796 consensus_queue.cc:260] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.404273 30793 consensus_queue.cc:260] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.411868 30899 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.415274 30899 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.417620 30899 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.418684 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:43.422598 30604 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:43.423784 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:43.424551 30753 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:43.426020 30793 ts_tablet_manager.cc:1434] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.035s user 0.009s sys 0.001s
I20251025 20:03:43.426858 30793 tablet_bootstrap.cc:492] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:43.428220 30796 ts_tablet_manager.cc:1434] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.036s user 0.004s sys 0.004s
I20251025 20:03:43.429114 30796 tablet_bootstrap.cc:492] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:43.425274 30676 raft_consensus.cc:2468] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:43.429756 30645 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:43.432184 30716 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:43.433146 30856 raft_consensus.cc:2804] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:43.435578 30500 catalog_manager.cc:5649] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:43.433658 30899 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:43.438323 30899 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:43.438709 30899 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.440068 30856 raft_consensus.cc:697] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:43.440999 30856 consensus_queue.cc:237] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.451354 30793 tablet_bootstrap.cc:654] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.451835 30868 consensus_queue.cc:1048] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:43.458148 30899 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.458611 30796 tablet_bootstrap.cc:654] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.470208 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:43.471184 30604 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.479043 30899 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.482079 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:43.483103 30752 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.500566 30500 catalog_manager.cc:5649] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:43.504242 30800 tablet_bootstrap.cc:492] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:43.507581 30800 ts_tablet_manager.cc:1403] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.167s user 0.017s sys 0.010s
I20251025 20:03:43.517956 30604 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:43.521656 30814 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.522351 30814 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.524291 30645 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:43.524473 30814 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.525872 30834 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:43.520887 30800 raft_consensus.cc:359] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.526510 30800 raft_consensus.cc:385] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.526515 30834 raft_consensus.cc:697] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:43.526839 30800 raft_consensus.cc:740] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.527529 30800 consensus_queue.cc:260] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.527644 30834 consensus_queue.cc:237] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.532824 30793 tablet_bootstrap.cc:492] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:43.533352 30793 ts_tablet_manager.cc:1403] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.107s user 0.016s sys 0.028s
W20251025 20:03:43.538028 30716 consensus_peers.cc:597] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e -> Peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Couldn't send request to peer 583af532dff04e8ebba45013323d516d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251025 20:03:43.533229 30645 consensus_peers.cc:597] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:43.540457 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:43.542866 30645 leader_election.cc:343] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.544430 30800 ts_tablet_manager.cc:1434] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.027s user 0.005s sys 0.002s
I20251025 20:03:43.544786 30603 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.528035 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:43.547032 30752 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
W20251025 20:03:43.546993 30645 leader_election.cc:343] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.547689 30645 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:43.549427 30814 raft_consensus.cc:2749] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:43.545518 30800 tablet_bootstrap.cc:492] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:43.552152 30793 raft_consensus.cc:359] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.541282 30856 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.553180 30793 raft_consensus.cc:385] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.553570 30793 raft_consensus.cc:740] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.553717 30856 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.554497 30793 consensus_queue.cc:260] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.564373 30793 ts_tablet_manager.cc:1434] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.031s user 0.005s sys 0.005s
I20251025 20:03:43.565232 30800 tablet_bootstrap.cc:654] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.556559 30502 catalog_manager.cc:5649] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:43.566401 30793 tablet_bootstrap.cc:492] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:43.575074 30793 tablet_bootstrap.cc:654] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.595403 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:43.597294 30719 leader_election.cc:343] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.598171 30815 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.615528 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:43.628854 30800 tablet_bootstrap.cc:492] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
W20251025 20:03:43.629338 30716 leader_election.cc:343] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.628862 30796 tablet_bootstrap.cc:492] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:43.630005 30800 ts_tablet_manager.cc:1403] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.085s user 0.031s sys 0.008s
W20251025 20:03:43.616708 30645 consensus_peers.cc:597] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:43.630378 30796 ts_tablet_manager.cc:1403] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.201s user 0.027s sys 0.011s
I20251025 20:03:43.634722 30716 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:43.636497 30859 raft_consensus.cc:2749] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:43.630788 30815 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.637933 30796 raft_consensus.cc:359] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.638620 30796 raft_consensus.cc:385] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.638938 30796 raft_consensus.cc:740] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.640444 30815 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:43.640285 30796 consensus_queue.cc:260] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.643561 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:43.643954 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:43.645102 30716 leader_election.cc:343] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:43.645661 30719 leader_election.cc:343] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.646234 30719 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:43.648882 30796 ts_tablet_manager.cc:1434] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20251025 20:03:43.649782 30796 tablet_bootstrap.cc:492] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:43.650118 30814 consensus_queue.cc:1048] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:43.651036 30815 raft_consensus.cc:2749] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:43.654426 30800 raft_consensus.cc:359] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.631053 30856 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:43.655169 30800 raft_consensus.cc:385] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.655509 30800 raft_consensus.cc:740] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.656411 30800 consensus_queue.cc:260] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.657696 30807 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:43.658253 30807 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.661293 30800 ts_tablet_manager.cc:1434] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20251025 20:03:43.662323 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:43.663626 30800 tablet_bootstrap.cc:492] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:43.664160 30796 tablet_bootstrap.cc:654] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.672183 30807 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.674526 30800 tablet_bootstrap.cc:654] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.673717 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:43.679567 30570 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:43.680558 30793 tablet_bootstrap.cc:492] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:43.680644 30807 raft_consensus.cc:2749] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:43.681157 30793 ts_tablet_manager.cc:1403] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.115s user 0.018s sys 0.013s
I20251025 20:03:43.682224 30814 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.682753 30814 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.684070 30793 raft_consensus.cc:359] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.684551 30793 raft_consensus.cc:385] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.684728 30793 raft_consensus.cc:740] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.684952 30814 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.685144 30793 consensus_queue.cc:260] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.688247 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:43.689515 30793 ts_tablet_manager.cc:1434] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.008s user 0.007s sys 0.002s
W20251025 20:03:43.691118 30645 leader_election.cc:343] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.689241 30902 consensus_queue.cc:1048] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:43.691766 30793 tablet_bootstrap.cc:492] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:43.692510 30902 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.692991 30902 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.695158 30902 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.689263 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:43.697983 30753 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:43.698989 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:43.699369 30645 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:43.700560 30814 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251025 20:03:43.700616 30567 leader_election.cc:343] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.701032 30814 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:43.701422 30814 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.703445 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:43.704494 30753 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:43.705657 30570 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:43.706490 30902 raft_consensus.cc:2804] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:43.706859 30902 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:43.707276 30902 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:43.715600 30796 tablet_bootstrap.cc:492] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:43.716189 30796 ts_tablet_manager.cc:1403] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.067s user 0.011s sys 0.010s
I20251025 20:03:43.718626 30793 tablet_bootstrap.cc:654] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.719615 30800 tablet_bootstrap.cc:492] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:43.720119 30800 ts_tablet_manager.cc:1403] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.012s
I20251025 20:03:43.721138 30796 raft_consensus.cc:359] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.721868 30796 raft_consensus.cc:385] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.722189 30796 raft_consensus.cc:740] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.726971 30796 consensus_queue.cc:260] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.729221 30796 ts_tablet_manager.cc:1434] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.013s user 0.005s sys 0.004s
I20251025 20:03:43.731000 30796 tablet_bootstrap.cc:492] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:43.733217 30856 consensus_queue.cc:1048] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:43.739873 30800 raft_consensus.cc:359] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.742478 30796 tablet_bootstrap.cc:654] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.784247 30800 raft_consensus.cc:385] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 20:03:43.789247 30567 consensus_peers.cc:597] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 -> Peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Couldn't send request to peer 583af532dff04e8ebba45013323d516d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:43.801784 30800 raft_consensus.cc:740] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.802600 30800 consensus_queue.cc:260] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.801265 30856 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.817139 30856 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.796353 30796 tablet_bootstrap.cc:492] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:43.821061 30796 ts_tablet_manager.cc:1403] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.090s user 0.011s sys 0.005s
I20251025 20:03:43.825260 30796 raft_consensus.cc:359] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.826315 30796 raft_consensus.cc:385] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:43.828953 30856 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:43.831300 30815 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.832019 30815 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.837328 30815 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:43.838601 30800 ts_tablet_manager.cc:1434] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.118s user 0.009s sys 0.000s
I20251025 20:03:43.826691 30796 raft_consensus.cc:740] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:43.839954 30796 consensus_queue.cc:260] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.842482 30796 ts_tablet_manager.cc:1434] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.021s user 0.008s sys 0.001s
I20251025 20:03:43.843580 30796 tablet_bootstrap.cc:492] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:43.849341 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:43.850858 30719 leader_election.cc:343] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:43.851091 30796 tablet_bootstrap.cc:654] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.806973 30814 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.860560 30814 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:43.861393 30800 tablet_bootstrap.cc:492] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:43.862154 30935 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:43.862732 30935 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.876252 30800 tablet_bootstrap.cc:654] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:43.877848 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:43.800976 30902 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:43.865172 30935 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:44.111621 30902 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:44.218281 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:44.221401 30834 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:44.225844 30793 tablet_bootstrap.cc:492] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:44.226447 30793 ts_tablet_manager.cc:1403] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.535s user 0.079s sys 0.128s
I20251025 20:03:44.229029 30796 tablet_bootstrap.cc:492] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:44.229435 30796 ts_tablet_manager.cc:1403] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.386s user 0.081s sys 0.113s
I20251025 20:03:44.227155 30868 consensus_queue.cc:1048] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251025 20:03:44.232729 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:44.233330 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:44.233446 30677 raft_consensus.cc:2468] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:44.233875 30796 raft_consensus.cc:359] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.234584 30796 raft_consensus.cc:385] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:44.234889 30796 raft_consensus.cc:740] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
W20251025 20:03:44.234918 30567 leader_election.cc:343] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.235757 30796 consensus_queue.cc:260] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.223320 30834 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.239954 30796 ts_tablet_manager.cc:1434] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.010s user 0.002s sys 0.005s
I20251025 20:03:44.241194 30834 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:44.241585 30716 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:44.242121 30800 tablet_bootstrap.cc:492] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:44.242607 30800 ts_tablet_manager.cc:1403] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.386s user 0.031s sys 0.069s
I20251025 20:03:44.243019 30793 raft_consensus.cc:359] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.243407 30815 raft_consensus.cc:2804] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:44.243780 30793 raft_consensus.cc:385] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:44.244032 30815 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:44.244092 30793 raft_consensus.cc:740] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:44.244473 30815 raft_consensus.cc:3060] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:44.242568 30716 leader_election.cc:343] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.244954 30793 consensus_queue.cc:260] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.245435 30800 raft_consensus.cc:359] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.245766 30716 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:44.246187 30800 raft_consensus.cc:385] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:44.246534 30800 raft_consensus.cc:740] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:44.246582 30856 raft_consensus.cc:2749] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:44.247402 30800 consensus_queue.cc:260] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.248136 30793 ts_tablet_manager.cc:1434] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20251025 20:03:44.249106 30793 tablet_bootstrap.cc:492] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
W20251025 20:03:44.249310 30570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.004s sys 0.007s
W20251025 20:03:44.249667 30570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.004s sys 0.007s
I20251025 20:03:44.249758 30800 ts_tablet_manager.cc:1434] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251025 20:03:44.250698 30800 tablet_bootstrap.cc:492] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:44.254181 30807 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:44.254763 30807 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.256880 30807 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:44.257476 30800 tablet_bootstrap.cc:654] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:44.257205 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:44.258118 30750 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:44.259469 30942 consensus_queue.cc:1048] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:44.259933 30645 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:44.265457 30942 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:44.266014 30942 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:44.266362 30942 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:44.271799 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:03:44.281590 30604 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.000s sys 0.005s
W20251025 20:03:44.282183 30604 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s user 0.001s sys 0.005s
W20251025 20:03:44.287801 30719 leader_election.cc:343] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.304101 30793 tablet_bootstrap.cc:654] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:44.314337 30814 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:44.316349 30814 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.318704 30814 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:44.329236 30800 tablet_bootstrap.cc:492] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:44.330024 30800 ts_tablet_manager.cc:1403] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.080s user 0.029s sys 0.028s
I20251025 20:03:44.330139 30796 tablet_bootstrap.cc:492] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:44.322610 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:44.388772 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:44.394846 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 45304us
I20251025 20:03:44.360225 30815 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.524353 30814 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:44.525224 30814 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.505416 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:44.536921 30807 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:44.538882 30807 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:44.542320 30645 leader_election.cc:343] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.541515 30807 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:44.570995 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:44.575654 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 261038us
I20251025 20:03:44.579694 30597 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:44.579727 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 257147us
I20251025 20:03:44.580837 30942 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.592465 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:44.593425 30800 raft_consensus.cc:359] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.594183 30800 raft_consensus.cc:385] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:44.594458 30800 raft_consensus.cc:740] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:44.595158 30800 consensus_queue.cc:260] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.602092 30800 ts_tablet_manager.cc:1434] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.272s user 0.007s sys 0.000s
W20251025 20:03:44.599733 30567 leader_election.cc:343] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.604346 30815 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:44.604511 30942 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:44.605118 30942 consensus_queue.cc:1048] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:44.606338 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:44.606441 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:44.607156 30674 raft_consensus.cc:3060] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:44.615654 30793 tablet_bootstrap.cc:492] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:44.616441 30793 ts_tablet_manager.cc:1403] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.368s user 0.013s sys 0.009s
I20251025 20:03:44.622296 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:03:44.623808 30719 leader_election.cc:343] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.635543 30796 tablet_bootstrap.cc:654] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:44.631183 30645 leader_election.cc:343] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:44.640749 30645 leader_election.cc:343] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.642711 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:44.646813 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:44.647538 30751 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:44.659140 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:44.660063 30674 raft_consensus.cc:2468] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:44.669677 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:44.660033 30752 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:44.660962 30953 consensus_queue.cc:1048] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20251025 20:03:44.666647 30751 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
W20251025 20:03:44.668591 30645 leader_election.cc:343] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.672302 30716 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
W20251025 20:03:44.672854 30567 leader_election.cc:343] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.680440 30942 consensus_queue.cc:1048] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 20:03:44.689854 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:44.697108 30752 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:44.714905 30676 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:44.725996 30645 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:44.727957 30645 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:44.729065 30835 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:44.729305 30645 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:44.729585 30835 raft_consensus.cc:697] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:44.730410 30835 consensus_queue.cc:237] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.731593 30954 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:44.732327 30954 raft_consensus.cc:697] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:44.733601 30954 consensus_queue.cc:237] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.734833 30800 tablet_bootstrap.cc:492] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:44.734956 30834 raft_consensus.cc:2749] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 20:03:44.744019 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 71831.8us
I20251025 20:03:44.747838 30793 raft_consensus.cc:359] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.748418 30793 raft_consensus.cc:385] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:44.748668 30793 raft_consensus.cc:740] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:44.749434 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:44.749223 30793 consensus_queue.cc:260] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.749737 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:44.750399 30955 raft_consensus.cc:2804] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
W20251025 20:03:44.751008 30570 leader_election.cc:343] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.751495 30570 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:44.752725 30935 raft_consensus.cc:2749] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:44.752854 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:03:44.753247 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 84218us
I20251025 20:03:44.753634 30748 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:44.754604 30570 leader_election.cc:343] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.755066 30570 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:44.755524 30793 ts_tablet_manager.cc:1434] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.139s user 0.007s sys 0.006s
I20251025 20:03:44.756246 30935 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:44.756590 30793 tablet_bootstrap.cc:492] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
W20251025 20:03:44.758800 30719 consensus_peers.cc:597] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:44.760914 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:44.762022 30955 raft_consensus.cc:697] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:44.763048 30955 consensus_queue.cc:237] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.827247 30570 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:44.837414 30807 raft_consensus.cc:2749] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:44.849946 30814 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:44.854039 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:44.870343 30645 leader_election.cc:343] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:44.853775 30748 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:44.871106 30645 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:44.873299 30570 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:44.872817 30834 raft_consensus.cc:2749] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:44.878777 30796 tablet_bootstrap.cc:492] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:44.879348 30796 ts_tablet_manager.cc:1403] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.550s user 0.027s sys 0.000s
I20251025 20:03:44.880849 30902 raft_consensus.cc:2804] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:44.881366 30902 raft_consensus.cc:697] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:44.882177 30800 tablet_bootstrap.cc:654] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:44.882416 30796 raft_consensus.cc:359] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.882287 30902 consensus_queue.cc:237] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.883055 30796 raft_consensus.cc:385] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:44.883353 30796 raft_consensus.cc:740] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:44.884006 30796 consensus_queue.cc:260] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.886454 30796 ts_tablet_manager.cc:1434] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251025 20:03:44.887488 30796 tablet_bootstrap.cc:492] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:44.894119 30796 tablet_bootstrap.cc:654] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:44.901516 30800 tablet_bootstrap.cc:492] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:44.902101 30800 ts_tablet_manager.cc:1403] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.167s user 0.017s sys 0.024s
I20251025 20:03:44.904994 30800 raft_consensus.cc:359] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.905766 30800 raft_consensus.cc:385] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:44.906088 30800 raft_consensus.cc:740] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:44.906747 30800 consensus_queue.cc:260] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.909091 30800 ts_tablet_manager.cc:1434] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251025 20:03:44.910220 30800 tablet_bootstrap.cc:492] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:44.915486 30793 tablet_bootstrap.cc:654] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:44.926017 30800 tablet_bootstrap.cc:654] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:44.928915 30603 raft_consensus.cc:3060] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:44.934468 30501 catalog_manager.cc:5649] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:44.941641 30796 tablet_bootstrap.cc:492] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:44.942239 30796 ts_tablet_manager.cc:1403] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.055s user 0.017s sys 0.012s
I20251025 20:03:44.941412 30501 catalog_manager.cc:5649] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:44.948261 30796 raft_consensus.cc:359] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.949306 30796 raft_consensus.cc:385] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:44.949678 30796 raft_consensus.cc:740] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:44.950644 30796 consensus_queue.cc:260] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.955788 30796 ts_tablet_manager.cc:1434] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.013s user 0.010s sys 0.000s
I20251025 20:03:44.956743 30796 tablet_bootstrap.cc:492] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:44.964828 30796 tablet_bootstrap.cc:654] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:44.973330 30793 tablet_bootstrap.cc:492] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:44.974268 30793 ts_tablet_manager.cc:1403] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.218s user 0.012s sys 0.005s
I20251025 20:03:44.980118 30793 raft_consensus.cc:359] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.981415 30793 raft_consensus.cc:385] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:44.981849 30793 raft_consensus.cc:740] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:44.983038 30793 consensus_queue.cc:260] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:44.986951 30793 ts_tablet_manager.cc:1434] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.012s user 0.007s sys 0.001s
I20251025 20:03:44.988423 30793 tablet_bootstrap.cc:492] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:45.001010 30796 tablet_bootstrap.cc:492] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:45.001605 30800 tablet_bootstrap.cc:492] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:45.001638 30796 ts_tablet_manager.cc:1403] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.014s
I20251025 20:03:45.002321 30800 ts_tablet_manager.cc:1403] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.092s user 0.028s sys 0.028s
I20251025 20:03:45.004760 30793 tablet_bootstrap.cc:654] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:45.005251 30800 raft_consensus.cc:359] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.005510 30796 raft_consensus.cc:359] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.006103 30800 raft_consensus.cc:385] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:45.006374 30796 raft_consensus.cc:385] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:45.006515 30800 raft_consensus.cc:740] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:45.006743 30796 raft_consensus.cc:740] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:45.007376 30800 consensus_queue.cc:260] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.007524 30796 consensus_queue.cc:260] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.009573 30902 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.010155 30902 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.010582 30807 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.010936 30796 ts_tablet_manager.cc:1434] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20251025 20:03:45.011072 30807 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.012030 30796 tablet_bootstrap.cc:492] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:45.013136 30807 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.014400 30856 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.015043 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:45.014917 30856 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.015731 30676 raft_consensus.cc:2468] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:45.017189 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:45.018797 30567 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:45.019842 30856 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:45.017491 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:45.021579 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:45.021811 30570 leader_election.cc:343] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.019764 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:45.023237 30570 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:45.023645 30719 leader_election.cc:343] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.024233 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:45.024475 30807 raft_consensus.cc:2804] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:45.024629 30935 raft_consensus.cc:2749] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:45.025164 30807 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:45.017769 30902 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:45.026628 30716 leader_election.cc:343] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.026650 30807 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.028389 30856 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.027215 30716 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:45.028928 30856 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.029867 30800 ts_tablet_manager.cc:1434] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.027s user 0.007s sys 0.002s
I20251025 20:03:45.031955 30800 tablet_bootstrap.cc:492] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:45.033288 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:45.036319 30796 tablet_bootstrap.cc:654] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:45.029976 30955 raft_consensus.cc:2749] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 20:03:45.038393 30719 leader_election.cc:343] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.050709 30807 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.053368 30807 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.054919 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:45.055807 30675 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.058828 30793 tablet_bootstrap.cc:492] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:45.059223 30793 ts_tablet_manager.cc:1403] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.007s
I20251025 20:03:45.061841 30793 raft_consensus.cc:359] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.062597 30793 raft_consensus.cc:385] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:45.062862 30793 raft_consensus.cc:740] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:45.063512 30793 consensus_queue.cc:260] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.066071 30793 ts_tablet_manager.cc:1434] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251025 20:03:45.068025 30793 tablet_bootstrap.cc:492] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:45.067679 30856 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:45.067734 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:03:45.073019 30570 leader_election.cc:343] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.075762 30675 raft_consensus.cc:2468] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:45.076586 30800 tablet_bootstrap.cc:654] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:45.078532 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:45.080739 30716 leader_election.cc:343] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.081270 30716 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:45.084852 30567 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:45.086568 30807 raft_consensus.cc:2804] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:45.087088 30807 raft_consensus.cc:697] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:45.085237 30856 raft_consensus.cc:2749] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:45.087956 30807 consensus_queue.cc:237] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.102348 30502 catalog_manager.cc:5649] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:45.116535 30796 tablet_bootstrap.cc:492] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:45.117308 30796 ts_tablet_manager.cc:1403] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.106s user 0.014s sys 0.007s
I20251025 20:03:45.120555 30796 raft_consensus.cc:359] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.121483 30796 raft_consensus.cc:385] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:45.125674 30793 tablet_bootstrap.cc:654] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:45.127044 30499 catalog_manager.cc:5649] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:45.121836 30796 raft_consensus.cc:740] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:45.134753 30796 consensus_queue.cc:260] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.140277 30800 tablet_bootstrap.cc:492] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:45.140853 30800 ts_tablet_manager.cc:1403] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.010s
I20251025 20:03:45.143776 30796 ts_tablet_manager.cc:1434] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.026s user 0.007s sys 0.003s
I20251025 20:03:45.144815 30796 tablet_bootstrap.cc:492] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:45.145344 30955 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.146018 30955 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.146447 30800 raft_consensus.cc:359] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.147193 30800 raft_consensus.cc:385] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:45.147454 30800 raft_consensus.cc:740] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:45.148284 30800 consensus_queue.cc:260] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.150429 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:45.150190 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:45.151311 30676 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:45.152323 30800 ts_tablet_manager.cc:1434] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20251025 20:03:45.152566 30716 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:45.153349 30800 tablet_bootstrap.cc:492] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
W20251025 20:03:45.153489 30719 leader_election.cc:343] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.153429 30856 raft_consensus.cc:2804] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:45.154109 30856 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:45.154505 30856 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.155740 30868 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.156215 30868 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.156450 30793 tablet_bootstrap.cc:492] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:45.157089 30793 ts_tablet_manager.cc:1403] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.089s user 0.010s sys 0.008s
I20251025 20:03:45.158265 30868 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.158310 30955 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:45.159989 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:45.214177 30793 raft_consensus.cc:359] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.215020 30793 raft_consensus.cc:385] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:45.215327 30793 raft_consensus.cc:740] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:45.161154 30598 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.216060 30793 consensus_queue.cc:260] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.160079 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:45.218596 30793 ts_tablet_manager.cc:1434] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.061s user 0.012s sys 0.022s
I20251025 20:03:45.219671 30793 tablet_bootstrap.cc:492] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:45.214213 30750 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:45.226049 30834 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.226384 30796 tablet_bootstrap.cc:654] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:45.226652 30834 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.230051 30834 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.168725 30856 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.231987 30834 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.236227 30834 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.238337 30834 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.248220 30800 tablet_bootstrap.cc:654] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:45.250633 30815 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.253450 30815 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.233868 30645 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:45.258235 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:45.259063 30815 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:45.263657 30719 leader_election.cc:343] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.264258 30793 tablet_bootstrap.cc:654] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:45.264165 30868 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:45.266112 30868 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:03:45.266093 30645 leader_election.cc:343] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.268790 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:45.273324 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:45.273351 30719 leader_election.cc:343] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.270499 30868 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.267930 30856 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:45.280541 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:45.282958 30796 tablet_bootstrap.cc:492] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:45.283227 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:45.283491 30676 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:45.283635 30796 ts_tablet_manager.cc:1403] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.139s user 0.037s sys 0.024s
I20251025 20:03:45.283933 30675 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.287690 30796 raft_consensus.cc:359] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.283257 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:45.291049 30796 raft_consensus.cc:385] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:45.291198 30750 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:45.291443 30796 raft_consensus.cc:740] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:45.294881 30796 consensus_queue.cc:260] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.295720 30675 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:45.298583 30796 ts_tablet_manager.cc:1434] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20251025 20:03:45.300941 30716 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:45.302558 30856 raft_consensus.cc:2804] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:45.301731 30499 catalog_manager.cc:5649] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:45.302943 30856 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:45.310067 30856 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.309059 30796 tablet_bootstrap.cc:492] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:45.313973 30645 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:45.314808 30835 raft_consensus.cc:2804] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:45.315191 30835 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:45.315538 30835 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.321834 30796 tablet_bootstrap.cc:654] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:45.321957 30716 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:45.323303 30815 raft_consensus.cc:2804] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:45.323908 30815 raft_consensus.cc:697] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:45.323937 30835 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.324879 30902 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.325394 30902 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.327728 30902 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.329034 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:45.329967 30750 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:45.331521 30570 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:45.332434 30902 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:45.333865 30902 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:45.303309 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:45.305367 30834 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.308226 30603 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.308226 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:45.334399 30902 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:45.334066 30815 consensus_queue.cc:237] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.288734 30868 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.337077 30834 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.341562 30942 consensus_queue.cc:1048] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:45.342340 30955 consensus_queue.cc:1048] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:45.344621 30868 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:45.348712 30645 leader_election.cc:343] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:45.349807 30645 leader_election.cc:343] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:45.350708 30645 consensus_peers.cc:597] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:45.351645 30814 consensus_queue.cc:1048] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:45.355160 30835 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.334915 30856 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.358453 30834 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.361389 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:45.362610 30793 tablet_bootstrap.cc:492] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:45.363370 30793 ts_tablet_manager.cc:1403] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.144s user 0.022s sys 0.005s
W20251025 20:03:45.363595 30719 leader_election.cc:343] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.364496 30800 tablet_bootstrap.cc:492] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:45.365036 30800 ts_tablet_manager.cc:1403] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.212s user 0.015s sys 0.000s
W20251025 20:03:45.366657 30645 leader_election.cc:343] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.367357 30645 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:45.375337 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:45.376057 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:45.376240 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:45.378299 30793 raft_consensus.cc:359] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.379853 30793 raft_consensus.cc:385] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:45.380285 30793 raft_consensus.cc:740] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:45.382679 30796 tablet_bootstrap.cc:492] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:45.383250 30796 ts_tablet_manager.cc:1403] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.075s user 0.016s sys 0.034s
I20251025 20:03:45.387511 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:45.387691 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:45.389017 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:45.373346 30835 raft_consensus.cc:2749] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:45.389539 30902 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.394649 30793 consensus_queue.cc:260] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.397354 30501 catalog_manager.cc:5649] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:45.408619 30800 raft_consensus.cc:359] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.400686 30750 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.409541 30800 raft_consensus.cc:385] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:45.410332 30800 raft_consensus.cc:740] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:45.411227 30800 consensus_queue.cc:260] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.412600 30674 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.403990 30752 raft_consensus.cc:2393] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:45.403350 30856 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:45.432204 30796 raft_consensus.cc:359] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.433072 30796 raft_consensus.cc:385] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:45.433362 30796 raft_consensus.cc:740] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:45.434124 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:45.438177 30807 consensus_queue.cc:1048] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:45.440392 30796 consensus_queue.cc:260] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:45.419111 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 49943.3us
I20251025 20:03:45.420120 30600 raft_consensus.cc:1240] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20251025 20:03:45.421223 30800 ts_tablet_manager.cc:1434] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.056s user 0.010s sys 0.001s
I20251025 20:03:45.422201 30748 raft_consensus.cc:2393] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:45.422693 30597 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:45.444940 30800 tablet_bootstrap.cc:492] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:45.447259 30793 ts_tablet_manager.cc:1434] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.084s user 0.006s sys 0.004s
W20251025 20:03:45.447731 30719 consensus_peers.cc:597] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:45.448444 30793 tablet_bootstrap.cc:492] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:45.452402 30645 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:45.459211 30645 leader_election.cc:343] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.462991 30868 consensus_queue.cc:1059] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:45.466789 30597 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:45.467537 30935 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.468475 30834 raft_consensus.cc:3055] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:45.468132 30935 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.489428 30796 ts_tablet_manager.cc:1434] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.097s user 0.007s sys 0.001s
I20251025 20:03:45.493947 30645 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:45.499732 30796 tablet_bootstrap.cc:492] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:45.491753 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:45.469022 30834 raft_consensus.cc:740] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:45.495971 30814 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.512157 30814 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.514335 30814 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.514870 30814 consensus_queue.cc:1048] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:45.500609 30942 raft_consensus.cc:2749] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:03:45.499732 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:45.510974 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:45.516955 30679 raft_consensus.cc:2393] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:45.511545 30676 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:45.520270 30834 consensus_queue.cc:260] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.473753 30935 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.492947 30902 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.530606 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:45.532675 30716 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:45.533039 30835 raft_consensus.cc:2749] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:45.533360 30567 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:45.534204 30902 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:45.534700 30902 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:45.535130 30902 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:45.529414 30834 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:45.542030 30815 raft_consensus.cc:2749] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:03:45.542210 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:45.559098 30645 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:45.560995 30814 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:45.560983 30750 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:45.561818 30814 raft_consensus.cc:697] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:45.562000 30935 consensus_queue.cc:1048] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:45.566789 30570 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:45.567281 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:45.567657 30935 raft_consensus.cc:2749] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:45.567520 30793 tablet_bootstrap.cc:654] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:45.565797 30814 consensus_queue.cc:237] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.580704 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:45.611521 30800 tablet_bootstrap.cc:654] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:45.612768 30501 catalog_manager.cc:5649] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:45.617300 30645 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:45.619401 30835 raft_consensus.cc:2749] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:45.646117 30902 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:45.648162 30774 debug-util.cc:398] Leaking SignalData structure 0x7b0800041640 after lost signal to thread 29446
W20251025 20:03:45.695809 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 70962.7us
I20251025 20:03:45.750633 30815 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.819538 30815 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:45.760213 30999 log.cc:927] Time spent T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20251025 20:03:45.851431 30834 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:45.898483 30834 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:45.901317 30834 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:45.902892 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:45.910053 30815 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:45.934123 30645 leader_election.cc:343] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:45.944545 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:45.946884 31000 log.cc:927] Time spent T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.261s user 0.017s sys 0.015s
W20251025 20:03:45.963943 30968 log.cc:927] Time spent T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.340s user 0.006s sys 0.036s
W20251025 20:03:45.970248 30993 log.cc:927] Time spent T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.225s user 0.004s sys 0.013s
W20251025 20:03:45.971463 30774 debug-util.cc:398] Leaking SignalData structure 0x7b080012c4a0 after lost signal to thread 30536
I20251025 20:03:45.974718 30793 tablet_bootstrap.cc:492] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:45.976559 30793 ts_tablet_manager.cc:1403] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.528s user 0.017s sys 0.010s
W20251025 20:03:45.977551 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 340197us
W20251025 20:03:45.991176 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 227289us
I20251025 20:03:46.011394 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:46.005576 30793 raft_consensus.cc:359] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.018533 30793 raft_consensus.cc:385] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.018932 30793 raft_consensus.cc:740] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.019980 30793 consensus_queue.cc:260] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.022130 31004 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.022871 31004 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:46.000702 30774 debug-util.cc:398] Leaking SignalData structure 0x7b080013eb40 after lost signal to thread 30628
W20251025 20:03:46.047089 30774 debug-util.cc:398] Leaking SignalData structure 0x7b08001765c0 after lost signal to thread 30703
I20251025 20:03:46.047170 30902 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:46.048451 30774 debug-util.cc:398] Leaking SignalData structure 0x7b0800149840 after lost signal to thread 30777
I20251025 20:03:46.032946 30674 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.053433 30793 ts_tablet_manager.cc:1434] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.076s user 0.010s sys 0.000s
I20251025 20:03:46.040583 31004 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:46.049527 30902 consensus_queue.cc:1048] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20251025 20:03:46.055306 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:46.058285 30800 tablet_bootstrap.cc:492] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:46.058838 30800 ts_tablet_manager.cc:1403] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.614s user 0.025s sys 0.048s
I20251025 20:03:46.066699 30796 tablet_bootstrap.cc:654] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:46.072810 30719 leader_election.cc:343] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.079455 30793 tablet_bootstrap.cc:492] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:46.081008 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:46.061748 30800 raft_consensus.cc:359] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.099864 30800 raft_consensus.cc:385] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.100173 30800 raft_consensus.cc:740] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.100754 30800 consensus_queue.cc:260] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.103157 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:46.103940 30676 raft_consensus.cc:2468] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:46.105836 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:46.106381 30752 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.107573 30597 raft_consensus.cc:1240] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:46.109004 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:46.109606 30745 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:46.110165 30716 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:46.111614 30716 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:46.114218 30815 raft_consensus.cc:2804] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:46.114590 30815 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:46.114897 30815 raft_consensus.cc:3060] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:46.118702 30645 leader_election.cc:343] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.119725 30645 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:46.125558 31004 consensus_queue.cc:1048] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251025 20:03:46.130949 30942 consensus_queue.cc:1059] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:46.133626 30796 tablet_bootstrap.cc:492] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:46.132790 30800 ts_tablet_manager.cc:1434] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.074s user 0.007s sys 0.000s
I20251025 20:03:46.134895 30796 ts_tablet_manager.cc:1403] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.640s user 0.035s sys 0.018s
I20251025 20:03:46.135459 30800 tablet_bootstrap.cc:492] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:46.133214 30942 raft_consensus.cc:3055] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:46.136444 30942 raft_consensus.cc:740] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:46.137234 30942 consensus_queue.cc:260] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.138433 30942 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:46.126227 30814 raft_consensus.cc:2749] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 20:03:46.143381 30719 consensus_peers.cc:597] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:46.152843 30796 raft_consensus.cc:359] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.153652 30796 raft_consensus.cc:385] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.153942 30796 raft_consensus.cc:740] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.154693 30796 consensus_queue.cc:260] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.167372 30955 raft_consensus.cc:2749] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:46.170540 30796 ts_tablet_manager.cc:1434] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.035s user 0.004s sys 0.004s
I20251025 20:03:46.171675 30796 tablet_bootstrap.cc:492] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:46.171975 31012 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.172513 31012 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.176518 30745 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:46.177218 31012 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.177476 30748 raft_consensus.cc:1240] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:46.178128 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:46.179343 30815 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:46.181674 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:46.182262 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:03:46.183928 30719 leader_election.cc:343] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.184108 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:46.186481 30815 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:46.188567 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:46.198642 30679 raft_consensus.cc:3060] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.192589 30570 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:46.206441 30796 tablet_bootstrap.cc:654] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.218266 31012 raft_consensus.cc:2749] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:46.219720 30955 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.220250 30955 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.222441 30793 tablet_bootstrap.cc:654] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.220920 31012 consensus_queue.cc:1048] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:46.224159 30955 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:46.224426 30679 raft_consensus.cc:2468] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:46.228065 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:46.228502 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:46.230335 30716 leader_election.cc:343] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:46.230989 30719 leader_election.cc:343] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.232437 30719 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:46.232744 30570 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:46.233695 31012 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:46.234282 31012 raft_consensus.cc:697] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:46.235260 31012 consensus_queue.cc:237] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.237583 30955 raft_consensus.cc:2749] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:46.238736 30716 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:46.240231 30955 raft_consensus.cc:2804] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:46.241756 30955 raft_consensus.cc:697] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:46.242949 30955 consensus_queue.cc:237] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.281236 30800 tablet_bootstrap.cc:654] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.296904 30501 catalog_manager.cc:5649] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 1 to 2, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
W20251025 20:03:46.303123 30645 consensus_peers.cc:597] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20251025 20:03:46.323594 30800 tablet_bootstrap.cc:492] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:46.324189 30800 ts_tablet_manager.cc:1403] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.189s user 0.012s sys 0.008s
I20251025 20:03:46.327484 30796 tablet_bootstrap.cc:492] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:46.328080 30796 ts_tablet_manager.cc:1403] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.157s user 0.012s sys 0.014s
I20251025 20:03:46.332252 30796 raft_consensus.cc:359] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.333256 30796 raft_consensus.cc:385] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.333470 30793 tablet_bootstrap.cc:492] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:46.333550 30796 raft_consensus.cc:740] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.334177 30793 ts_tablet_manager.cc:1403] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.266s user 0.021s sys 0.012s
I20251025 20:03:46.333783 30502 catalog_manager.cc:5649] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:46.334605 30796 consensus_queue.cc:260] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.338040 30793 raft_consensus.cc:359] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.338794 30793 raft_consensus.cc:385] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.339074 30793 raft_consensus.cc:740] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.339732 30793 consensus_queue.cc:260] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.341925 30793 ts_tablet_manager.cc:1434] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251025 20:03:46.342944 30793 tablet_bootstrap.cc:492] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:46.347404 30800 raft_consensus.cc:359] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.349929 30800 raft_consensus.cc:385] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.350301 30800 raft_consensus.cc:740] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.351080 30800 consensus_queue.cc:260] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.359122 30793 tablet_bootstrap.cc:654] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.360170 30796 ts_tablet_manager.cc:1434] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.027s user 0.007s sys 0.003s
I20251025 20:03:46.366493 30800 ts_tablet_manager.cc:1434] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.042s user 0.006s sys 0.003s
I20251025 20:03:46.364785 30796 tablet_bootstrap.cc:492] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:46.367705 30800 tablet_bootstrap.cc:492] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:46.377465 30800 tablet_bootstrap.cc:654] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.389210 30800 tablet_bootstrap.cc:492] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:46.391031 30800 ts_tablet_manager.cc:1403] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.007s
I20251025 20:03:46.396347 30597 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.398075 30796 tablet_bootstrap.cc:654] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.394752 30800 raft_consensus.cc:359] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.404139 30800 raft_consensus.cc:385] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.404383 30800 raft_consensus.cc:740] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.404965 30800 consensus_queue.cc:260] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.409945 30800 ts_tablet_manager.cc:1434] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20251025 20:03:46.410959 30800 tablet_bootstrap.cc:492] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:46.416268 30603 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.418587 30800 tablet_bootstrap.cc:654] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.427704 30793 tablet_bootstrap.cc:492] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:46.428328 30793 ts_tablet_manager.cc:1403] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.016s
I20251025 20:03:46.428265 30815 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.429010 30815 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.433312 30815 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:46.433413 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:46.434559 30793 raft_consensus.cc:359] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.435284 30793 raft_consensus.cc:385] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.435335 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:46.435659 30793 raft_consensus.cc:740] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.436439 30793 consensus_queue.cc:260] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:46.436937 30716 leader_election.cc:343] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.439394 30793 ts_tablet_manager.cc:1434] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20251025 20:03:46.441040 30793 tablet_bootstrap.cc:492] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
W20251025 20:03:46.443337 30719 leader_election.cc:343] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.444083 30719 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:46.445883 30815 raft_consensus.cc:2749] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:46.452098 30796 tablet_bootstrap.cc:492] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:46.452647 30796 ts_tablet_manager.cc:1403] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.091s user 0.015s sys 0.004s
I20251025 20:03:46.452972 30793 tablet_bootstrap.cc:654] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.456691 30800 tablet_bootstrap.cc:492] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:46.457216 30800 ts_tablet_manager.cc:1403] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.005s
I20251025 20:03:46.461004 30800 raft_consensus.cc:359] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.461866 30800 raft_consensus.cc:385] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.462213 30800 raft_consensus.cc:740] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.462894 30800 consensus_queue.cc:260] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.465358 30800 ts_tablet_manager.cc:1434] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20251025 20:03:46.469599 30800 tablet_bootstrap.cc:492] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:46.467960 30796 raft_consensus.cc:359] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.474133 30796 raft_consensus.cc:385] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.474463 30796 raft_consensus.cc:740] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.475239 30796 consensus_queue.cc:260] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.476118 30800 tablet_bootstrap.cc:654] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.478674 30807 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.479833 30807 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.482123 30807 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.483110 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:46.477792 30796 ts_tablet_manager.cc:1434] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.025s user 0.000s sys 0.005s
I20251025 20:03:46.484035 30674 raft_consensus.cc:2393] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:46.484826 30796 tablet_bootstrap.cc:492] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:46.486495 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:46.487432 30745 raft_consensus.cc:2393] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:46.492049 30570 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:46.495610 30807 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.497521 30796 tablet_bootstrap.cc:654] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.499339 30793 tablet_bootstrap.cc:492] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:46.499877 30793 ts_tablet_manager.cc:1403] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.006s
I20251025 20:03:46.503083 30793 raft_consensus.cc:359] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.503856 30793 raft_consensus.cc:385] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.504154 30793 raft_consensus.cc:740] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.504860 30793 consensus_queue.cc:260] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.507216 30793 ts_tablet_manager.cc:1434] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251025 20:03:46.508293 30793 tablet_bootstrap.cc:492] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:46.509698 30800 tablet_bootstrap.cc:492] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:46.510236 30800 ts_tablet_manager.cc:1403] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.002s
I20251025 20:03:46.512967 30800 raft_consensus.cc:359] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.513862 30800 raft_consensus.cc:385] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.514179 30800 raft_consensus.cc:740] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.514835 30793 tablet_bootstrap.cc:654] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.514909 30800 consensus_queue.cc:260] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.518482 30800 ts_tablet_manager.cc:1434] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20251025 20:03:46.521335 30793 tablet_bootstrap.cc:492] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:46.521880 30793 ts_tablet_manager.cc:1403] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20251025 20:03:46.523213 30800 tablet_bootstrap.cc:492] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:46.522780 30807 raft_consensus.cc:2749] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:46.541036 30800 tablet_bootstrap.cc:654] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.544581 30793 raft_consensus.cc:359] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.548233 30793 raft_consensus.cc:385] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.548614 30793 raft_consensus.cc:740] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.549413 30793 consensus_queue.cc:260] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.551868 30942 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.552387 30942 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.553786 31012 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.554284 31012 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.554669 30942 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.555716 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:46.557109 31012 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.558498 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:46.558799 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:46.559185 30745 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:46.552181 30793 ts_tablet_manager.cc:1434] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20251025 20:03:46.560421 30645 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:03:46.561916 30645 leader_election.cc:343] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.562659 30793 tablet_bootstrap.cc:492] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:46.562950 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:46.564428 30942 raft_consensus.cc:2804] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:46.564853 30942 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:46.565212 30942 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.566807 30995 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.567318 30995 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.570772 30570 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:46.572177 31012 raft_consensus.cc:2749] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:46.574069 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:46.575053 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:46.575471 30645 leader_election.cc:343] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.576195 30995 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:46.578408 30645 leader_election.cc:343] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.578835 30645 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:46.580462 30814 raft_consensus.cc:2749] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:46.581709 30793 tablet_bootstrap.cc:654] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.595199 30800 tablet_bootstrap.cc:492] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:46.595593 30796 tablet_bootstrap.cc:492] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:46.595844 30800 ts_tablet_manager.cc:1403] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.022s
I20251025 20:03:46.596087 30796 ts_tablet_manager.cc:1403] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.023s
I20251025 20:03:46.601807 30796 raft_consensus.cc:359] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.602597 30796 raft_consensus.cc:385] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.602896 30796 raft_consensus.cc:740] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.603255 30800 raft_consensus.cc:359] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.603500 30796 consensus_queue.cc:260] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.604054 30800 raft_consensus.cc:385] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.604354 30800 raft_consensus.cc:740] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.605155 30800 consensus_queue.cc:260] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.609679 30800 ts_tablet_manager.cc:1434] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20251025 20:03:46.611083 30800 tablet_bootstrap.cc:492] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:46.614707 30796 ts_tablet_manager.cc:1434] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20251025 20:03:46.615866 30796 tablet_bootstrap.cc:492] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:46.619899 30793 tablet_bootstrap.cc:492] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:46.620501 30793 ts_tablet_manager.cc:1403] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.008s
I20251025 20:03:46.624256 30800 tablet_bootstrap.cc:654] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.620811 30942 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.626053 30793 raft_consensus.cc:359] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.629628 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:46.630174 30793 raft_consensus.cc:385] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.630489 30793 raft_consensus.cc:740] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.631426 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:46.631273 30793 consensus_queue.cc:260] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:46.631911 30645 leader_election.cc:343] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.632030 30745 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.634406 30793 ts_tablet_manager.cc:1434] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.014s user 0.009s sys 0.001s
I20251025 20:03:46.636857 30793 tablet_bootstrap.cc:492] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:46.634203 30942 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.652626 30793 tablet_bootstrap.cc:654] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.650110 30745 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:46.662833 30800 tablet_bootstrap.cc:492] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:46.663523 30807 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.663734 30800 ts_tablet_manager.cc:1403] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.053s user 0.019s sys 0.011s
I20251025 20:03:46.664080 30807 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.664863 30796 tablet_bootstrap.cc:654] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.666309 30807 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.666824 30800 raft_consensus.cc:359] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.667603 30800 raft_consensus.cc:385] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.667901 30800 raft_consensus.cc:740] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.668098 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:46.668783 30745 raft_consensus.cc:2468] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:46.668601 30800 consensus_queue.cc:260] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.669193 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:46.669963 30673 raft_consensus.cc:2468] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:46.664090 30645 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:46.671583 30800 ts_tablet_manager.cc:1434] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251025 20:03:46.673012 30942 raft_consensus.cc:2804] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:46.673672 30942 raft_consensus.cc:697] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:46.675475 30942 consensus_queue.cc:237] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.677197 30800 tablet_bootstrap.cc:492] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:46.680195 30796 tablet_bootstrap.cc:492] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:46.680681 30796 ts_tablet_manager.cc:1403] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.008s
I20251025 20:03:46.681666 30567 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:46.683468 30902 raft_consensus.cc:2804] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:46.683835 30902 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:46.684201 30902 raft_consensus.cc:3060] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.685388 30796 raft_consensus.cc:359] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.685956 30800 tablet_bootstrap.cc:654] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.686151 30796 raft_consensus.cc:385] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.686507 30796 raft_consensus.cc:740] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.687173 30796 consensus_queue.cc:260] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.687623 31012 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.688083 31012 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.689810 30796 ts_tablet_manager.cc:1434] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20251025 20:03:46.690829 30796 tablet_bootstrap.cc:492] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
W20251025 20:03:46.692132 30719 consensus_peers.cc:597] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:46.697152 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:46.697954 30673 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:46.702909 30567 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:46.704818 31012 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.705878 30796 tablet_bootstrap.cc:654] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.706261 31012 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:46.706632 31012 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:46.707015 31012 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.710988 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:46.711724 30745 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:46.723264 30800 tablet_bootstrap.cc:492] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:46.723857 30800 ts_tablet_manager.cc:1403] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.047s user 0.024s sys 0.008s
I20251025 20:03:46.729516 30995 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.730248 30995 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.731391 30902 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.734947 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:46.735149 30995 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.723839 31012 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:46.737576 30645 leader_election.cc:343] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.738631 30800 raft_consensus.cc:359] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.739663 30800 raft_consensus.cc:385] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.740011 30800 raft_consensus.cc:740] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.737576 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:46.740870 30800 consensus_queue.cc:260] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.744009 30800 ts_tablet_manager.cc:1434] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.020s user 0.005s sys 0.004s
I20251025 20:03:46.745682 30800 tablet_bootstrap.cc:492] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:46.741444 30752 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:46.755002 30645 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:46.757652 30835 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:46.758620 30835 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:46.758999 30835 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.759873 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:46.760540 30752 raft_consensus.cc:3060] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.758635 30902 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.766369 31012 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:46.754179 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 50538.1us
I20251025 20:03:46.768509 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:46.777299 30745 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.777880 31012 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.783741 31012 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.784793 30793 tablet_bootstrap.cc:492] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:46.784387 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:46.785362 30793 ts_tablet_manager.cc:1403] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.149s user 0.012s sys 0.006s
I20251025 20:03:46.785416 30674 raft_consensus.cc:3060] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.785465 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:46.786316 30673 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.787029 31012 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.789434 30752 raft_consensus.cc:2468] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:46.790524 30793 raft_consensus.cc:359] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.791281 30793 raft_consensus.cc:385] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.791095 30570 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:46.778249 30835 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.792419 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:46.793526 30796 tablet_bootstrap.cc:492] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:46.794083 30796 ts_tablet_manager.cc:1403] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.103s user 0.023s sys 0.014s
I20251025 20:03:46.791932 30793 raft_consensus.cc:740] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
W20251025 20:03:46.795089 30570 leader_election.cc:343] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.795176 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:46.795575 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:46.796283 30748 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:46.798305 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:46.798557 30796 raft_consensus.cc:359] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.799170 30674 raft_consensus.cc:2468] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:46.799355 30796 raft_consensus.cc:385] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:46.799638 30796 raft_consensus.cc:740] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:46.800328 30796 consensus_queue.cc:260] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.801648 30793 consensus_queue.cc:260] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:46.805763 30645 leader_election.cc:343] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:46.807032 30567 leader_election.cc:343] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:46.807514 30567 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:46.809288 31039 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:46.812005 30796 ts_tablet_manager.cc:1434] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20251025 20:03:46.813129 30796 tablet_bootstrap.cc:492] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:46.813416 30750 raft_consensus.cc:1275] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 20:03:46.815939 31039 consensus_queue.cc:1048] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:46.818442 30745 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:46.820180 30748 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:46.800238 30835 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.820959 30570 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:46.822312 30645 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:46.823771 30501 catalog_manager.cc:5649] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:46.825337 30793 ts_tablet_manager.cc:1434] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.040s user 0.008s sys 0.000s
I20251025 20:03:46.826799 30793 tablet_bootstrap.cc:492] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:46.802837 31012 raft_consensus.cc:2804] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:46.824568 31039 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:46.826926 30995 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:46.831996 31039 raft_consensus.cc:697] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:46.831403 30995 raft_consensus.cc:697] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:46.833818 31039 consensus_queue.cc:237] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.834806 30995 consensus_queue.cc:237] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.839063 30673 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:46.840893 30814 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.841442 30814 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.845372 30814 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.846755 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:46.847754 30597 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:03:46.848816 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:46.849431 30674 raft_consensus.cc:1275] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20251025 20:03:46.850052 30674 pending_rounds.cc:85] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 0
I20251025 20:03:46.850464 30674 pending_rounds.cc:107] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20251025 20:03:46.850889 30674 raft_consensus.cc:2889] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:03:46.849383 30645 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:46.831070 31012 raft_consensus.cc:697] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:46.855762 31012 consensus_queue.cc:237] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.858110 30835 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:46.858707 30835 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:46.859166 30835 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:46.868451 30800 tablet_bootstrap.cc:654] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.895489 31012 consensus_queue.cc:1048] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:46.883323 30902 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.903172 30902 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.905380 30793 tablet_bootstrap.cc:654] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.873622 30807 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.918287 30815 consensus_queue.cc:1048] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:46.909044 30902 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.947986 30796 tablet_bootstrap.cc:654] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:46.948956 30902 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:46.916498 30807 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:46.953495 30902 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:46.950465 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 66954.7us
I20251025 20:03:46.984297 30902 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:46.992313 30992 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:47.000327 30603 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.000579 30992 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.002910 30992 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:46.988566 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:47.011967 30807 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:47.014735 30815 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:47.017670 30502 catalog_manager.cc:5649] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:47.015455 30815 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.040612 30815 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:47.039815 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:47.037950 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 53606.7us
I20251025 20:03:47.048630 31048 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:47.060811 30499 catalog_manager.cc:5649] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:47.067658 30793 tablet_bootstrap.cc:492] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:47.068143 30793 ts_tablet_manager.cc:1403] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.242s user 0.026s sys 0.011s
I20251025 20:03:47.076628 30793 raft_consensus.cc:359] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.078372 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:47.081881 30793 raft_consensus.cc:385] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:47.084120 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:47.061492 31048 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.086975 31048 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:47.069265 31043 log.cc:927] Time spent T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20251025 20:03:47.082278 30793 raft_consensus.cc:740] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:47.088757 30793 consensus_queue.cc:260] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.090297 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:47.099393 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:47.100694 30601 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:47.103533 31049 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:47.104368 31049 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.106940 30835 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.107762 31049 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:47.109244 30719 leader_election.cc:343] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.109794 30597 raft_consensus.cc:3060] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.112752 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:47.115428 30800 tablet_bootstrap.cc:492] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:47.115970 30800 ts_tablet_manager.cc:1403] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.371s user 0.017s sys 0.029s
I20251025 20:03:47.118100 30793 ts_tablet_manager.cc:1434] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.050s user 0.007s sys 0.003s
I20251025 20:03:47.118481 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:47.119132 30600 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:47.120498 30796 tablet_bootstrap.cc:492] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:47.120399 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:47.121024 30796 ts_tablet_manager.cc:1403] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.308s user 0.013s sys 0.003s
I20251025 20:03:47.121312 30675 raft_consensus.cc:2368] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:47.119158 30793 tablet_bootstrap.cc:492] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:47.114607 30502 catalog_manager.cc:5649] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:47.129205 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:47.134133 30719 leader_election.cc:343] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.136416 30719 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:47.136700 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:47.137669 30750 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:47.139755 31049 raft_consensus.cc:2804] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:47.140199 31049 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:47.140569 31049 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:03:47.142454 30567 leader_election.cc:343] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.146320 30835 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:47.148165 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:47.151103 30679 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:47.150108 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:47.152259 30750 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 3
W20251025 20:03:47.152935 30570 leader_election.cc:343] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.153028 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:47.153494 30570 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:47.153656 30600 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:47.150146 30796 raft_consensus.cc:359] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.155251 30796 raft_consensus.cc:385] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:47.155493 30796 raft_consensus.cc:740] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:47.156083 30796 consensus_queue.cc:260] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.154323 30902 raft_consensus.cc:2749] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:47.176079 30796 ts_tablet_manager.cc:1434] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.055s user 0.007s sys 0.002s
I20251025 20:03:47.177016 30796 tablet_bootstrap.cc:492] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:47.177690 30800 raft_consensus.cc:359] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.178344 30800 raft_consensus.cc:385] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:47.178565 30800 raft_consensus.cc:740] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:47.179760 30800 consensus_queue.cc:260] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.176262 30942 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:03:47.173863 30716 leader_election.cc:343] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.185159 30800 ts_tablet_manager.cc:1434] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.069s user 0.008s sys 0.000s
I20251025 20:03:47.186619 30800 tablet_bootstrap.cc:492] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:47.184685 30942 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.196972 30942 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:47.188575 30716 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:47.199093 30955 raft_consensus.cc:2749] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 20:03:47.199625 30716 leader_election.cc:343] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.200197 30716 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:47.207073 30955 raft_consensus.cc:2749] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:47.207587 30750 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:03:47.216645 30716 leader_election.cc:400] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975) with higher term. Message: Invalid argument: T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:47.216336 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:47.217911 30716 leader_election.cc:403] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:03:47.218281 30745 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:47.219125 30955 raft_consensus.cc:2749] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975) with higher term. Message: Invalid argument: T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:47.222502 30603 raft_consensus.cc:3060] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.226243 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:47.229282 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:47.229960 30674 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:47.225569 30570 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:47.237720 31012 raft_consensus.cc:2804] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:47.238124 31012 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:47.238452 31012 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:47.240631 30645 consensus_peers.cc:597] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:47.242079 31048 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:47.242623 31048 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.247565 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:47.251742 30719 leader_election.cc:343] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.250020 31048 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:47.253931 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:47.260187 30793 tablet_bootstrap.cc:654] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:47.235365 30600 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:03:47.263406 31049 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.271682 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:47.272429 30602 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:47.269312 30835 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:47.280206 30567 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:47.291994 30800 tablet_bootstrap.cc:654] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:47.298192 30645 leader_election.cc:343] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.285185 30835 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.304514 30835 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:47.294118 31046 raft_consensus.cc:2749] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:47.292774 31049 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:47.309549 30645 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:47.314678 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:47.316949 30679 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:47.315572 30995 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:47.319840 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:47.320592 30748 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:47.317788 30995 raft_consensus.cc:697] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:47.324193 30995 consensus_queue.cc:237] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.326759 31012 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.328250 30645 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:47.330457 30942 raft_consensus.cc:2804] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:47.330878 30942 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:47.331279 30942 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.333482 30796 tablet_bootstrap.cc:654] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:47.347635 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:47.348392 30678 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.349561 30835 consensus_queue.cc:1048] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20251025 20:03:47.372308 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:47.377664 31047 consensus_queue.cc:1048] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:47.393277 30992 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:47.393940 30992 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.396186 30992 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:47.397534 30716 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:47.400924 31049 raft_consensus.cc:2804] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:47.401355 31049 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:47.401841 31049 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.402580 30645 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:47.389339 31012 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:47.403517 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:47.411885 30814 raft_consensus.cc:2804] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:47.412910 30814 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:47.414695 30814 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.422061 30800 tablet_bootstrap.cc:492] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:47.422781 30800 ts_tablet_manager.cc:1403] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.236s user 0.017s sys 0.014s
I20251025 20:03:47.425573 30942 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.425988 30800 raft_consensus.cc:359] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.427229 30800 raft_consensus.cc:385] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:47.427657 30800 raft_consensus.cc:740] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:47.428469 30800 consensus_queue.cc:260] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.430433 30800 ts_tablet_manager.cc:1434] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251025 20:03:47.430794 30793 tablet_bootstrap.cc:492] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:47.431546 30793 ts_tablet_manager.cc:1403] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.313s user 0.029s sys 0.027s
I20251025 20:03:47.431702 30800 tablet_bootstrap.cc:492] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:47.440222 30793 raft_consensus.cc:359] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.441044 30793 raft_consensus.cc:385] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:47.441316 30793 raft_consensus.cc:740] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:47.431955 31049 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.442409 30793 consensus_queue.cc:260] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.432507 30814 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.446944 31049 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:47.492586 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:47.494959 30603 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:47.496516 30602 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:47.497161 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:47.417608 30674 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:47.506138 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:47.507354 30750 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.446496 30942 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:47.522444 30796 tablet_bootstrap.cc:492] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:47.522737 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:47.529417 30719 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:47.531842 30992 raft_consensus.cc:2804] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:47.532038 30814 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:47.532507 30992 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:47.533226 30992 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.544067 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:47.545497 30676 raft_consensus.cc:2393] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:47.545789 30796 ts_tablet_manager.cc:1403] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.369s user 0.014s sys 0.013s
I20251025 20:03:47.546089 30719 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:47.550112 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:47.554205 31047 consensus_queue.cc:1048] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:47.555341 30793 ts_tablet_manager.cc:1434] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.123s user 0.006s sys 0.003s
I20251025 20:03:47.557865 31048 raft_consensus.cc:2804] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:47.558715 31048 raft_consensus.cc:697] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:47.560557 31048 consensus_queue.cc:237] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.567875 31047 consensus_queue.cc:1048] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:47.557792 30793 tablet_bootstrap.cc:492] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:47.561762 30800 tablet_bootstrap.cc:654] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:47.578598 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 175883us
W20251025 20:03:47.582015 30719 leader_election.cc:343] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.583019 30796 raft_consensus.cc:359] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.583984 30796 raft_consensus.cc:385] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:47.584306 30796 raft_consensus.cc:740] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:47.589089 30796 consensus_queue.cc:260] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.591290 30678 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:47.601058 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:47.602238 30751 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.603322 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:47.604135 30674 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:03:47.615206 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 45287.8us
I20251025 20:03:47.630088 30567 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:47.631810 31039 raft_consensus.cc:2804] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
W20251025 20:03:47.637666 30716 leader_election.cc:343] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.639268 30716 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:47.632423 31039 raft_consensus.cc:697] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:47.641115 30955 raft_consensus.cc:2749] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:03:47.641532 31039 consensus_queue.cc:237] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.649973 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:47.649755 30902 consensus_queue.cc:1048] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:47.656342 30796 ts_tablet_manager.cc:1434] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.106s user 0.009s sys 0.001s
W20251025 20:03:47.661815 30645 leader_election.cc:343] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:47.666767 30645 leader_election.cc:343] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.672335 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:47.677364 31069 consensus_queue.cc:1048] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:47.679158 30992 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.683337 30992 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:47.689666 30793 tablet_bootstrap.cc:654] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:47.696163 30796 tablet_bootstrap.cc:492] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
W20251025 20:03:47.705106 30645 leader_election.cc:343] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.709393 30745 raft_consensus.cc:2393] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:47.710089 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:47.714488 30645 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:47.714745 30716 leader_election.cc:343] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.716114 30750 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:47.716761 30835 raft_consensus.cc:2749] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:03:47.720919 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:47.721688 30602 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.737810 30800 tablet_bootstrap.cc:492] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:47.743309 30800 ts_tablet_manager.cc:1403] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.312s user 0.018s sys 0.014s
I20251025 20:03:47.749342 30751 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:47.754099 30800 raft_consensus.cc:359] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.754993 30800 raft_consensus.cc:385] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:47.755290 30800 raft_consensus.cc:740] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:47.737488 30502 catalog_manager.cc:5649] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:47.755993 30800 consensus_queue.cc:260] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:47.767170 30645 consensus_peers.cc:597] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:47.772382 30800 ts_tablet_manager.cc:1434] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.029s user 0.007s sys 0.002s
I20251025 20:03:47.773837 30800 tablet_bootstrap.cc:492] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:47.778371 30796 tablet_bootstrap.cc:654] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:47.780959 30645 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:47.782330 30995 raft_consensus.cc:2804] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:47.782845 30995 raft_consensus.cc:697] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:47.783089 30501 catalog_manager.cc:5649] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:47.784096 30800 tablet_bootstrap.cc:654] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:47.783952 30995 consensus_queue.cc:237] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.791514 30602 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:47.797389 30719 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:47.799374 30992 raft_consensus.cc:2804] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:47.800043 30992 raft_consensus.cc:697] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:47.801126 30992 consensus_queue.cc:237] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.806625 30793 tablet_bootstrap.cc:492] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:47.807225 30793 ts_tablet_manager.cc:1403] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.250s user 0.018s sys 0.005s
I20251025 20:03:47.819263 30942 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:47.820011 30942 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.824023 30942 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:47.805334 30499 catalog_manager.cc:5649] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 2, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:47.826046 30597 raft_consensus.cc:1275] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:47.826089 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:47.832366 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:47.833227 30750 raft_consensus.cc:2468] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:47.834069 31039 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:47.834664 31039 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:47.843631 30645 leader_election.cc:343] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.848785 30645 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:47.849771 30942 consensus_queue.cc:1048] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:47.850610 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:47.851114 31039 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:47.852382 30567 leader_election.cc:343] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.852301 31069 raft_consensus.cc:2804] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:47.853088 31069 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:47.854233 31069 raft_consensus.cc:3060] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:47.855162 30800 tablet_bootstrap.cc:492] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:47.855702 30800 ts_tablet_manager.cc:1403] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.082s user 0.018s sys 0.015s
I20251025 20:03:47.856973 30793 raft_consensus.cc:359] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.858211 30793 raft_consensus.cc:385] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:47.858449 30793 raft_consensus.cc:740] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:47.861038 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:47.858958 30793 consensus_queue.cc:260] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:47.865374 30570 leader_election.cc:343] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.865964 30570 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:47.867385 31047 raft_consensus.cc:2749] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:47.872542 30796 tablet_bootstrap.cc:492] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:47.873291 30796 ts_tablet_manager.cc:1403] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.178s user 0.007s sys 0.018s
I20251025 20:03:47.931560 30793 ts_tablet_manager.cc:1434] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.124s user 0.024s sys 0.011s
I20251025 20:03:47.932751 30793 tablet_bootstrap.cc:492] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:47.932237 30796 raft_consensus.cc:359] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.933063 30750 raft_consensus.cc:1275] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:47.933164 30796 raft_consensus.cc:385] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:47.933588 30796 raft_consensus.cc:740] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:47.934347 30796 consensus_queue.cc:260] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.937842 30796 ts_tablet_manager.cc:1434] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.064s user 0.018s sys 0.017s
I20251025 20:03:47.942466 30995 consensus_queue.cc:1048] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:47.943365 30796 tablet_bootstrap.cc:492] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:47.942028 30800 raft_consensus.cc:359] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.943931 30800 raft_consensus.cc:385] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:47.944221 30800 raft_consensus.cc:740] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:47.944931 30800 consensus_queue.cc:260] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.951445 30800 ts_tablet_manager.cc:1434] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.094s user 0.015s sys 0.016s
I20251025 20:03:47.952746 30800 tablet_bootstrap.cc:492] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:47.941363 31069 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.976392 30793 tablet_bootstrap.cc:654] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:47.980240 30796 tablet_bootstrap.cc:654] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:47.985522 31048 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:47.986187 31048 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:47.988536 31048 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:47.989704 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:47.989970 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:47.990587 30676 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
W20251025 20:03:47.991433 30719 leader_election.cc:343] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:47.992841 30716 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:47.994074 30992 raft_consensus.cc:2804] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:47.994452 30992 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:47.994810 30992 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.000893 30800 tablet_bootstrap.cc:654] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.004460 31069 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.004319 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:48.005411 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:03:48.006098 30645 leader_election.cc:343] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.006222 31069 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.006757 31069 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.006094 30752 raft_consensus.cc:3060] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.009409 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:48.010666 30602 raft_consensus.cc:2468] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:48.011955 30645 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:48.012740 30942 raft_consensus.cc:2804] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:48.013119 30942 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:48.013434 30942 raft_consensus.cc:3060] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.019439 30752 raft_consensus.cc:2468] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:48.023123 30645 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:48.024015 30995 raft_consensus.cc:2804] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:48.024526 30995 raft_consensus.cc:697] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:48.025410 30995 consensus_queue.cc:237] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.033972 30602 raft_consensus.cc:1275] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 20:03:48.036653 30676 raft_consensus.cc:1275] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 20:03:48.036590 31073 consensus_queue.cc:1048] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:48.045154 30992 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.048537 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:48.042531 30942 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:48.052719 30719 leader_election.cc:343] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.054363 31073 consensus_queue.cc:1048] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:48.057956 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:48.059523 30645 leader_election.cc:343] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.060931 31069 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.064549 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:48.065232 30597 raft_consensus.cc:3060] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.061524 30502 catalog_manager.cc:5649] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:48.071027 30796 tablet_bootstrap.cc:492] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:48.071722 30796 ts_tablet_manager.cc:1403] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.129s user 0.011s sys 0.012s
I20251025 20:03:48.074015 30597 raft_consensus.cc:2468] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:48.074599 30992 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:48.075832 30645 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:48.076292 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:48.076891 30678 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.074950 30796 raft_consensus.cc:359] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.082096 30796 raft_consensus.cc:385] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.082481 30796 raft_consensus.cc:740] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.083263 31071 raft_consensus.cc:2804] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:48.083289 30796 consensus_queue.cc:260] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.083940 31071 raft_consensus.cc:697] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:48.084883 31071 consensus_queue.cc:237] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.086283 30796 ts_tablet_manager.cc:1434] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20251025 20:03:48.087430 30796 tablet_bootstrap.cc:492] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:48.087970 30793 tablet_bootstrap.cc:492] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:48.091599 30793 ts_tablet_manager.cc:1403] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.159s user 0.016s sys 0.010s
I20251025 20:03:48.092139 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:03:48.096907 30645 leader_election.cc:343] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.090764 30942 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.102135 30796 tablet_bootstrap.cc:654] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.103582 30793 raft_consensus.cc:359] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.113765 30793 raft_consensus.cc:385] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.114665 30793 raft_consensus.cc:740] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.115903 30793 consensus_queue.cc:260] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.119939 30793 ts_tablet_manager.cc:1434] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.028s user 0.009s sys 0.000s
I20251025 20:03:48.117064 31047 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.122776 31047 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.105849 30800 tablet_bootstrap.cc:492] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:48.124312 30800 ts_tablet_manager.cc:1403] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.172s user 0.009s sys 0.005s
I20251025 20:03:48.127267 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:48.127194 30800 raft_consensus.cc:359] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.127960 30800 raft_consensus.cc:385] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.127838 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.128317 30800 raft_consensus.cc:740] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.129041 30800 consensus_queue.cc:260] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.116385 30678 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:48.130502 30567 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:48.130920 31047 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.132135 31047 raft_consensus.cc:2749] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:48.133565 30716 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:48.133884 30796 tablet_bootstrap.cc:492] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:48.134161 30800 ts_tablet_manager.cc:1434] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20251025 20:03:48.134397 30796 ts_tablet_manager.cc:1403] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.008s
I20251025 20:03:48.134765 31048 raft_consensus.cc:2804] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:48.135394 30800 tablet_bootstrap.cc:492] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:48.135340 31048 raft_consensus.cc:697] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:48.136858 31048 consensus_queue.cc:237] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.142199 30793 tablet_bootstrap.cc:492] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:48.143270 30955 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.142827 30796 raft_consensus.cc:359] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.144582 30796 raft_consensus.cc:385] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.144822 30796 raft_consensus.cc:740] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.145359 30796 consensus_queue.cc:260] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.143761 30955 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.148267 30955 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:48.150655 31047 consensus_queue.cc:1048] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:48.151173 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:48.151206 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.152045 30602 raft_consensus.cc:2468] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:48.147873 30501 catalog_manager.cc:5649] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:48.159134 30719 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:03:48.160130 30716 leader_election.cc:343] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.160276 30992 raft_consensus.cc:2804] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:48.160840 30992 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:48.161289 30992 raft_consensus.cc:3060] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.179800 30602 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.180420 30796 ts_tablet_manager.cc:1434] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.046s user 0.008s sys 0.000s
I20251025 20:03:48.184299 30501 catalog_manager.cc:5649] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:48.194118 30796 tablet_bootstrap.cc:492] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:48.203157 30800 tablet_bootstrap.cc:654] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.207046 30678 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.247305 30992 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.269860 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:48.270717 30600 raft_consensus.cc:3060] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.290436 30992 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:48.292471 30796 tablet_bootstrap.cc:654] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.295723 31047 consensus_queue.cc:1048] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:48.296092 30955 consensus_queue.cc:1048] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:48.292585 30793 tablet_bootstrap.cc:654] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.303264 30499 catalog_manager.cc:5649] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:48.306372 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:03:48.309235 30716 leader_election.cc:343] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:48.318681 30645 consensus_peers.cc:597] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:48.341619 30796 tablet_bootstrap.cc:492] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:48.342201 30796 ts_tablet_manager.cc:1403] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.161s user 0.008s sys 0.009s
I20251025 20:03:48.344287 30502 catalog_manager.cc:5649] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:48.353103 30942 consensus_queue.cc:1048] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251025 20:03:48.354765 30600 raft_consensus.cc:2468] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:48.359767 30796 raft_consensus.cc:359] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.360395 30796 raft_consensus.cc:385] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.360699 30796 raft_consensus.cc:740] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.361058 30719 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:48.362577 30955 raft_consensus.cc:2804] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:48.363072 30955 raft_consensus.cc:697] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:48.363896 30955 consensus_queue.cc:237] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.361390 30796 consensus_queue.cc:260] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.371774 31047 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.372373 31047 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.374454 31047 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.376220 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.376763 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:48.377910 30567 leader_election.cc:343] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.378541 30800 tablet_bootstrap.cc:492] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
W20251025 20:03:48.378701 30570 leader_election.cc:343] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.379086 30800 ts_tablet_manager.cc:1403] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.244s user 0.040s sys 0.040s
I20251025 20:03:48.379267 30570 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:48.381647 30796 ts_tablet_manager.cc:1434] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.039s user 0.005s sys 0.001s
I20251025 20:03:48.382489 30793 tablet_bootstrap.cc:492] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:48.383045 30793 ts_tablet_manager.cc:1403] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.241s user 0.016s sys 0.018s
I20251025 20:03:48.383186 30796 tablet_bootstrap.cc:492] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:48.385143 31012 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:48.399607 31071 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.400274 31071 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.399652 31047 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.401671 31047 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.403360 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:48.403852 30793 raft_consensus.cc:359] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.404093 30800 raft_consensus.cc:359] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.404541 30793 raft_consensus.cc:385] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.404793 30800 raft_consensus.cc:385] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 20:03:48.404855 30645 leader_election.cc:343] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.405157 30800 raft_consensus.cc:740] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.405861 30800 consensus_queue.cc:260] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.407418 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:48.408305 30800 ts_tablet_manager.cc:1434] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.010s user 0.001s sys 0.004s
I20251025 20:03:48.409368 30800 tablet_bootstrap.cc:492] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:48.406239 31047 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.410392 31071 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.409437 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.424207 30796 tablet_bootstrap.cc:654] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.423806 31073 consensus_queue.cc:1048] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:48.404898 30793 raft_consensus.cc:740] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.427273 30793 consensus_queue.cc:260] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.430101 30793 ts_tablet_manager.cc:1434] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.047s user 0.008s sys 0.001s
I20251025 20:03:48.432850 30793 tablet_bootstrap.cc:492] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:48.435621 30800 tablet_bootstrap.cc:654] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:48.447484 30645 consensus_peers.cc:597] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:48.449358 30793 tablet_bootstrap.cc:654] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.451303 30567 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:48.452639 31047 raft_consensus.cc:2749] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:48.456804 31073 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.457489 31073 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.461263 31073 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:48.462528 30796 tablet_bootstrap.cc:492] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:48.463056 30796 ts_tablet_manager.cc:1403] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.005s
I20251025 20:03:48.465057 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:48.471288 30796 raft_consensus.cc:359] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.473009 30796 raft_consensus.cc:385] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.473299 30796 raft_consensus.cc:740] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.474658 30796 consensus_queue.cc:260] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:48.475215 30645 leader_election.cc:343] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.475721 30645 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:48.476639 30942 raft_consensus.cc:2749] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:48.476951 30796 ts_tablet_manager.cc:1434] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20251025 20:03:48.483718 30793 tablet_bootstrap.cc:492] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:48.484294 30793 ts_tablet_manager.cc:1403] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.007s
I20251025 20:03:48.487097 30793 raft_consensus.cc:359] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.487865 30793 raft_consensus.cc:385] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.488152 30793 raft_consensus.cc:740] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.488946 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.488797 30793 consensus_queue.cc:260] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.490052 30796 tablet_bootstrap.cc:492] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:48.495712 30800 tablet_bootstrap.cc:492] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:48.496263 30800 ts_tablet_manager.cc:1403] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.004s
I20251025 20:03:48.499467 30800 raft_consensus.cc:359] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.500075 30793 ts_tablet_manager.cc:1434] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20251025 20:03:48.500244 30800 raft_consensus.cc:385] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.501160 30800 raft_consensus.cc:740] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.501354 30793 tablet_bootstrap.cc:492] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:48.501808 30800 consensus_queue.cc:260] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.506273 30800 ts_tablet_manager.cc:1434] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20251025 20:03:48.508158 30800 tablet_bootstrap.cc:492] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:48.510337 30601 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.512854 30835 consensus_queue.cc:1048] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:48.515053 30796 tablet_bootstrap.cc:654] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.513039 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:48.518589 30835 consensus_queue.cc:1048] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:48.526041 30800 tablet_bootstrap.cc:654] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.537225 31050 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.537923 31050 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.543375 31050 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.545689 30719 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:48.543661 30793 tablet_bootstrap.cc:654] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.549443 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:48.553315 31048 raft_consensus.cc:2749] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 20:03:48.554282 30645 leader_election.cc:343] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.556643 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:48.558286 30645 leader_election.cc:343] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.558720 30645 leader_election.cc:304] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:48.554559 30502 catalog_manager.cc:5649] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:48.560156 30835 raft_consensus.cc:2749] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:48.577373 30750 raft_consensus.cc:3060] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.578117 31048 consensus_queue.cc:1048] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 20:03:48.580988 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:48.635728 30793 tablet_bootstrap.cc:492] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:48.636348 30793 ts_tablet_manager.cc:1403] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.135s user 0.008s sys 0.007s
I20251025 20:03:48.681962 30796 tablet_bootstrap.cc:492] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:48.682740 30796 ts_tablet_manager.cc:1403] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.205s user 0.016s sys 0.008s
I20251025 20:03:48.692020 30793 raft_consensus.cc:359] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.692937 30793 raft_consensus.cc:385] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.693221 30793 raft_consensus.cc:740] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.693886 30793 consensus_queue.cc:260] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.697846 30796 raft_consensus.cc:359] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.699069 31049 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.698679 30796 raft_consensus.cc:385] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.699607 31049 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.700047 30796 raft_consensus.cc:740] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.701485 30800 tablet_bootstrap.cc:492] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:48.700812 30796 consensus_queue.cc:260] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.701800 31049 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:48.702054 30800 ts_tablet_manager.cc:1403] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.195s user 0.028s sys 0.015s
I20251025 20:03:48.703732 30796 ts_tablet_manager.cc:1434] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20251025 20:03:48.696612 31050 consensus_queue.cc:1048] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251025 20:03:48.705060 30796 tablet_bootstrap.cc:492] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:48.717286 30793 ts_tablet_manager.cc:1434] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.081s user 0.008s sys 0.000s
W20251025 20:03:48.689085 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 57836.3us
I20251025 20:03:48.719476 30800 raft_consensus.cc:359] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.720340 30800 raft_consensus.cc:385] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.720651 30800 raft_consensus.cc:740] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.721323 30800 consensus_queue.cc:260] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.724330 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:48.725529 30800 ts_tablet_manager.cc:1434] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
I20251025 20:03:48.726828 30800 tablet_bootstrap.cc:492] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:48.719477 30793 tablet_bootstrap.cc:492] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
W20251025 20:03:48.728514 30719 consensus_peers.cc:597] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251025 20:03:48.730752 30719 leader_election.cc:343] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.737864 30807 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.738464 30807 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.740365 30807 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.744549 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.747607 31050 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:03:48.763027 30716 leader_election.cc:343] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.783911 30716 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:48.786361 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.766502 31050 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.789431 31050 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.774744 31067 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.790673 31048 raft_consensus.cc:2749] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:48.786206 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:48.790405 31067 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.795650 30570 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:48.796589 30807 raft_consensus.cc:2749] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:48.800474 31067 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:48.802452 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 43533.8us
I20251025 20:03:48.809934 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:48.811295 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:48.800567 31047 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:03:48.812642 30645 leader_election.cc:343] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.821647 30800 tablet_bootstrap.cc:654] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.821576 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:48.817236 31047 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.826128 31047 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.826874 30955 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.827378 30955 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.829387 30955 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:48.831387 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:48.832032 30598 raft_consensus.cc:2468] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:48.833177 30719 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:48.834123 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:48.834854 31049 raft_consensus.cc:2804] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:48.835291 31049 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:48.835546 30793 tablet_bootstrap.cc:654] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.835701 31049 raft_consensus.cc:3060] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:48.838828 30645 leader_election.cc:343] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.852250 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:48.852535 30645 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:48.854768 30796 tablet_bootstrap.cc:654] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:48.861053 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.866024 30942 raft_consensus.cc:2749] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:48.868142 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.834856 30751 raft_consensus.cc:2468] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
W20251025 20:03:48.874425 30645 leader_election.cc:343] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:48.874886 30567 leader_election.cc:343] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:48.876296 30716 consensus_peers.cc:597] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e -> Peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Couldn't send request to peer 583af532dff04e8ebba45013323d516d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251025 20:03:48.878712 30716 leader_election.cc:343] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.889750 30645 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:48.895318 30942 raft_consensus.cc:2804] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:48.895823 30942 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:48.895903 30600 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.896274 30942 raft_consensus.cc:3060] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.904143 31073 consensus_queue.cc:1048] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:48.906273 30942 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.907399 31049 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.909516 30752 raft_consensus.cc:2393] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:48.915894 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:48.916604 30595 raft_consensus.cc:3060] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.923053 30570 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:48.927188 30793 tablet_bootstrap.cc:492] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:48.928957 30800 tablet_bootstrap.cc:492] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:48.929514 30800 ts_tablet_manager.cc:1403] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.203s user 0.016s sys 0.011s
I20251025 20:03:48.929744 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:48.930596 31049 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:48.931460 30716 leader_election.cc:343] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.931612 30942 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.933456 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:48.934229 30752 raft_consensus.cc:3060] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.927683 30595 raft_consensus.cc:2468] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:48.936544 31122 raft_consensus.cc:2749] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:48.936831 30793 ts_tablet_manager.cc:1403] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.218s user 0.013s sys 0.007s
I20251025 20:03:48.939077 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:48.940089 31073 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.940591 31073 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:48.940932 30645 leader_election.cc:343] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.943523 30719 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:48.943915 30800 raft_consensus.cc:359] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.944655 30800 raft_consensus.cc:385] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.944730 30955 raft_consensus.cc:2804] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:48.945216 30800 raft_consensus.cc:740] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.946300 30800 consensus_queue.cc:260] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.945823 30955 raft_consensus.cc:697] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:48.948271 30955 consensus_queue.cc:237] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.949051 30800 ts_tablet_manager.cc:1434] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20251025 20:03:48.949616 30793 raft_consensus.cc:359] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.951201 30793 raft_consensus.cc:385] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.949052 30796 tablet_bootstrap.cc:492] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:48.956807 30793 raft_consensus.cc:740] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.955431 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.955433 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:48.957029 30796 ts_tablet_manager.cc:1403] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.252s user 0.015s sys 0.013s
W20251025 20:03:48.958649 30716 leader_election.cc:343] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:48.959913 30719 leader_election.cc:343] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.950249 30800 tablet_bootstrap.cc:492] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:48.960356 30719 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:48.961172 31048 raft_consensus.cc:2749] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:48.959513 30793 consensus_queue.cc:260] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.953433 31073 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:48.961156 30796 raft_consensus.cc:359] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.964187 30796 raft_consensus.cc:385] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:48.970191 30796 raft_consensus.cc:740] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:48.970320 30793 ts_tablet_manager.cc:1434] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.033s user 0.009s sys 0.000s
I20251025 20:03:48.971774 30793 tablet_bootstrap.cc:492] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:48.977183 31122 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.977766 31122 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:48.980254 31122 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:48.981537 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:48.981751 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:48.982744 30745 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
W20251025 20:03:48.983575 30567 leader_election.cc:343] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:48.985875 30570 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:48.987983 31122 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:48.988575 31122 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:48.989117 31122 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:48.990132 30800 tablet_bootstrap.cc:654] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.000499 31073 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:48.978281 30752 raft_consensus.cc:2468] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:49.001174 31073 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.014961 31073 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:49.016562 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:49.005846 30793 tablet_bootstrap.cc:654] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.019755 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:49.019412 30716 leader_election.cc:343] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.018823 30645 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:49.020471 30598 raft_consensus.cc:2468] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:49.021119 31124 raft_consensus.cc:2804] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:49.021603 31124 raft_consensus.cc:697] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:49.022478 31124 consensus_queue.cc:237] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.007190 31122 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.026937 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:48.970885 30796 consensus_queue.cc:260] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.028379 30800 tablet_bootstrap.cc:492] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
W20251025 20:03:49.028712 30567 leader_election.cc:343] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.029964 30719 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:49.029188 30501 catalog_manager.cc:5649] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:49.030969 30796 ts_tablet_manager.cc:1434] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.072s user 0.006s sys 0.001s
I20251025 20:03:49.032183 30796 tablet_bootstrap.cc:492] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:49.033986 31073 raft_consensus.cc:2804] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:49.034514 31073 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:49.034955 31073 raft_consensus.cc:3060] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.038551 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:49.040055 30745 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.029213 30800 ts_tablet_manager.cc:1403] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.079s user 0.015s sys 0.003s
I20251025 20:03:49.050450 31122 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.052033 30598 raft_consensus.cc:3060] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.059010 30796 tablet_bootstrap.cc:654] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.058403 30502 catalog_manager.cc:5649] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:49.055729 30800 raft_consensus.cc:359] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.060299 30800 raft_consensus.cc:385] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.060606 30800 raft_consensus.cc:740] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.064528 30800 consensus_queue.cc:260] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.065590 30745 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:49.067129 30570 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:49.067859 31073 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.068655 31122 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:49.071779 30800 ts_tablet_manager.cc:1434] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.021s user 0.004s sys 0.002s
I20251025 20:03:49.072654 30793 tablet_bootstrap.cc:492] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:49.073166 30793 ts_tablet_manager.cc:1403] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.102s user 0.013s sys 0.003s
I20251025 20:03:49.071322 31122 raft_consensus.cc:697] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:49.074223 31122 consensus_queue.cc:237] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.076318 30793 raft_consensus.cc:359] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.077200 30793 raft_consensus.cc:385] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.077484 30793 raft_consensus.cc:740] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.078416 30793 consensus_queue.cc:260] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.091329 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:49.092393 30800 tablet_bootstrap.cc:492] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:49.094030 31073 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:49.091785 30793 ts_tablet_manager.cc:1434] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.018s user 0.004s sys 0.003s
I20251025 20:03:49.092334 30600 raft_consensus.cc:3060] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.096630 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:49.098222 30793 tablet_bootstrap.cc:492] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
W20251025 20:03:49.101641 30716 leader_election.cc:343] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.104598 30796 tablet_bootstrap.cc:492] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:49.105227 30796 ts_tablet_manager.cc:1403] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.073s user 0.012s sys 0.002s
I20251025 20:03:49.109328 30796 raft_consensus.cc:359] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.111667 30796 raft_consensus.cc:385] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.112120 30796 raft_consensus.cc:740] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.113022 30796 consensus_queue.cc:260] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.118243 30796 ts_tablet_manager.cc:1434] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20251025 20:03:49.119335 30796 tablet_bootstrap.cc:492] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:49.109251 30600 raft_consensus.cc:2468] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:49.123109 30719 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:49.126243 30955 raft_consensus.cc:2804] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:49.128924 30955 raft_consensus.cc:697] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:49.139752 30955 consensus_queue.cc:237] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.150902 30800 tablet_bootstrap.cc:654] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.163582 30501 catalog_manager.cc:5649] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:49.167694 30796 tablet_bootstrap.cc:654] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.169059 30793 tablet_bootstrap.cc:654] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.181550 30800 tablet_bootstrap.cc:492] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:49.182217 30800 ts_tablet_manager.cc:1403] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.095s user 0.016s sys 0.000s
I20251025 20:03:49.185312 30800 raft_consensus.cc:359] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.190559 30800 raft_consensus.cc:385] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.190855 30800 raft_consensus.cc:740] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.191642 30800 consensus_queue.cc:260] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.197672 30800 ts_tablet_manager.cc:1434] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20251025 20:03:49.198876 30800 tablet_bootstrap.cc:492] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:49.211578 30502 catalog_manager.cc:5649] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:49.216771 30793 tablet_bootstrap.cc:492] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:49.217397 30793 ts_tablet_manager.cc:1403] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.119s user 0.021s sys 0.001s
I20251025 20:03:49.223444 30800 tablet_bootstrap.cc:654] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.224689 30793 raft_consensus.cc:359] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.225739 30793 raft_consensus.cc:385] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.226133 30793 raft_consensus.cc:740] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.227705 30796 tablet_bootstrap.cc:492] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:49.228214 30796 ts_tablet_manager.cc:1403] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.109s user 0.022s sys 0.004s
I20251025 20:03:49.231140 30796 raft_consensus.cc:359] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.231741 30796 raft_consensus.cc:385] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.232024 30796 raft_consensus.cc:740] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.230967 30793 consensus_queue.cc:260] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.233078 30796 consensus_queue.cc:260] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.235710 30796 ts_tablet_manager.cc:1434] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20251025 20:03:49.237977 30793 ts_tablet_manager.cc:1434] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.020s user 0.011s sys 0.000s
I20251025 20:03:49.239169 30793 tablet_bootstrap.cc:492] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:49.240614 30800 tablet_bootstrap.cc:492] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:49.243093 30796 tablet_bootstrap.cc:492] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:49.244809 30800 ts_tablet_manager.cc:1403] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.009s
I20251025 20:03:49.248087 30800 raft_consensus.cc:359] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.248781 30800 raft_consensus.cc:385] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.249058 30800 raft_consensus.cc:740] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.249774 31073 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.249653 30800 consensus_queue.cc:260] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.250349 31073 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.252187 30796 tablet_bootstrap.cc:654] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.253130 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:49.254647 30719 leader_election.cc:343] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.255044 30793 tablet_bootstrap.cc:654] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.255261 30800 ts_tablet_manager.cc:1434] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20251025 20:03:49.260098 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:49.261011 30676 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:49.263595 30800 tablet_bootstrap.cc:492] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:49.264062 30796 tablet_bootstrap.cc:492] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:49.263849 30716 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:49.264536 30796 ts_tablet_manager.cc:1403] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.004s
I20251025 20:03:49.264659 30942 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.265054 31049 raft_consensus.cc:2804] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:49.265139 30942 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.265538 31049 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:49.266552 31049 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.267242 30942 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.267551 30796 raft_consensus.cc:359] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.268316 30796 raft_consensus.cc:385] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.268574 30796 raft_consensus.cc:740] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.269196 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:49.269459 30796 consensus_queue.cc:260] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.270921 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:49.271696 30745 raft_consensus.cc:2393] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:49.271839 30796 ts_tablet_manager.cc:1434] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251025 20:03:49.272924 30796 tablet_bootstrap.cc:492] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:49.272907 30645 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:49.274382 30942 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.252408 31073 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:49.280939 30796 tablet_bootstrap.cc:654] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.282469 31050 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.283002 31050 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.285048 31050 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.287987 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:49.289378 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:49.290125 30745 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
W20251025 20:03:49.292714 30645 leader_election.cc:343] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.294390 31049 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.298306 30942 raft_consensus.cc:2749] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:49.299793 31049 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:49.288898 30800 tablet_bootstrap.cc:654] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.300835 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:49.303287 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:49.307541 30807 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.308360 30645 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:49.320703 30678 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.326156 30807 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.328948 30807 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:49.332489 30719 leader_election.cc:343] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.337530 31124 raft_consensus.cc:2804] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:49.343890 31124 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:49.350327 31124 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.343847 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:49.349253 30942 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.353235 30942 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.356022 30674 raft_consensus.cc:3060] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:49.359634 30570 leader_election.cc:343] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.354851 31048 consensus_queue.cc:1048] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20251025 20:03:49.363193 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:49.364078 30676 raft_consensus.cc:2468] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:49.365381 30567 leader_election.cc:304] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:49.368629 31139 raft_consensus.cc:2804] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:49.369196 31139 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:49.369663 31139 raft_consensus.cc:3060] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.375113 30942 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.377303 31049 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.378166 31049 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.381249 31049 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:49.384068 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:49.384215 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:49.386498 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:49.384778 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:49.385075 30796 tablet_bootstrap.cc:492] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
W20251025 20:03:49.387799 30716 leader_election.cc:343] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:49.388595 30719 leader_election.cc:343] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.388976 30796 ts_tablet_manager.cc:1403] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.116s user 0.010s sys 0.028s
I20251025 20:03:49.389367 30719 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:49.390468 31073 raft_consensus.cc:2749] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:49.391608 30793 tablet_bootstrap.cc:492] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:49.392110 30793 ts_tablet_manager.cc:1403] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.153s user 0.029s sys 0.006s
I20251025 20:03:49.389842 30745 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
W20251025 20:03:49.397173 30645 leader_election.cc:343] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.397258 30793 raft_consensus.cc:359] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.398463 30645 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:49.398723 30793 raft_consensus.cc:385] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.399176 30793 raft_consensus.cc:740] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.399971 30793 consensus_queue.cc:260] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.403013 30835 raft_consensus.cc:2804] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:49.403573 30835 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:49.403096 30796 raft_consensus.cc:359] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.403940 30835 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:49.404196 30796 raft_consensus.cc:385] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.404584 30796 raft_consensus.cc:740] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.407923 30800 tablet_bootstrap.cc:492] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:49.417711 30800 ts_tablet_manager.cc:1403] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.154s user 0.029s sys 0.011s
I20251025 20:03:49.411600 30793 ts_tablet_manager.cc:1434] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.019s user 0.006s sys 0.002s
I20251025 20:03:49.408438 30796 consensus_queue.cc:260] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.412810 31139 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.431923 30793 tablet_bootstrap.cc:492] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:49.405827 31124 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.444331 30796 ts_tablet_manager.cc:1434] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.055s user 0.008s sys 0.002s
I20251025 20:03:49.412696 30678 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:49.454808 30796 tablet_bootstrap.cc:492] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:49.448076 31071 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.462628 31124 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.467836 31139 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.463400 31071 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.468616 30793 tablet_bootstrap.cc:654] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.473764 30800 raft_consensus.cc:359] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.475401 30800 raft_consensus.cc:385] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.475584 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:49.476011 30800 raft_consensus.cc:740] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.475250 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:49.476202 31139 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:03:49.483395 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 59749.7us
I20251025 20:03:49.483675 30800 consensus_queue.cc:260] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.482571 30835 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.486516 30835 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.487593 30716 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
W20251025 20:03:49.488924 30570 leader_election.cc:343] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.492771 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:49.491125 31049 raft_consensus.cc:2804] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:49.492847 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:49.493695 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:49.493808 30751 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.493731 31049 raft_consensus.cc:697] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:49.495994 31049 consensus_queue.cc:237] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.483911 30676 raft_consensus.cc:3060] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:49.500407 30645 leader_election.cc:343] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.493116 30595 raft_consensus.cc:3060] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:49.501240 30645 leader_election.cc:343] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.483294 31071 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.503134 30800 ts_tablet_manager.cc:1434] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.084s user 0.004s sys 0.003s
I20251025 20:03:49.507020 30800 tablet_bootstrap.cc:492] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:49.482913 31139 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.531450 31139 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.542059 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:49.562610 30677 raft_consensus.cc:2468] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:49.565032 30678 raft_consensus.cc:3060] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.565392 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:49.567838 31139 consensus_queue.cc:1048] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:49.523162 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:49.549109 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:49.567847 30567 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:49.569648 30752 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:49.516726 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:49.569217 30602 raft_consensus.cc:2393] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:49.579180 31152 consensus_queue.cc:1048] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251025 20:03:49.580382 30676 raft_consensus.cc:2468] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
W20251025 20:03:49.549645 31151 log.cc:927] Time spent T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
W20251025 20:03:49.586422 30570 leader_election.cc:343] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.590204 31048 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.571985 31139 raft_consensus.cc:2804] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:49.596230 31048 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.604604 31139 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:03:49.615435 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:49.612668 30751 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:49.614645 31139 raft_consensus.cc:3060] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.620713 30796 tablet_bootstrap.cc:654] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.621299 30500 catalog_manager.cc:5649] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:03:49.612900 30567 leader_election.cc:304] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:49.626793 31122 raft_consensus.cc:2804] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:49.627532 31122 raft_consensus.cc:697] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:49.628728 31122 consensus_queue.cc:237] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.631458 31048 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:49.633431 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:49.634941 31047 consensus_queue.cc:1048] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 20:03:49.636605 30716 leader_election.cc:343] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.636897 30800 tablet_bootstrap.cc:654] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.611049 31049 consensus_queue.cc:1048] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20251025 20:03:49.639899 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:49.641341 30719 leader_election.cc:343] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:49.641817 30719 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:49.647883 30645 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:49.658159 31071 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.658738 31071 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.660809 31071 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.655611 30793 tablet_bootstrap.cc:492] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:49.661577 30807 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.662091 30793 ts_tablet_manager.cc:1403] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.236s user 0.015s sys 0.020s
I20251025 20:03:49.648772 31152 raft_consensus.cc:3060] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.654776 30645 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:49.664779 30807 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.643321 31049 raft_consensus.cc:2749] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:49.679371 31139 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.680024 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:49.673223 31071 raft_consensus.cc:2804] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:49.712972 30942 consensus_queue.cc:1048] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:49.712720 31071 raft_consensus.cc:697] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:49.723361 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:49.724478 31139 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.790050 30807 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.804378 30807 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:49.805023 30807 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.828979 30807 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:49.837481 30752 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:03:49.841251 30793 raft_consensus.cc:359] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.842357 30793 raft_consensus.cc:385] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.842710 30793 raft_consensus.cc:740] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.845022 30793 consensus_queue.cc:260] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.846212 30645 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:49.852691 30645 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:49.859431 30835 raft_consensus.cc:2804] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:49.859977 30835 raft_consensus.cc:697] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:49.862214 30942 raft_consensus.cc:2749] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:49.864287 30835 consensus_queue.cc:237] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.871390 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:49.835655 31071 consensus_queue.cc:237] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.892829 30793 ts_tablet_manager.cc:1434] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.230s user 0.009s sys 0.000s
W20251025 20:03:49.895211 31157 log.cc:927] Time spent T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20251025 20:03:49.896524 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:49.901424 31152 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:49.896593 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:49.877148 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:49.903436 30674 raft_consensus.cc:2393] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:49.912137 30796 tablet_bootstrap.cc:492] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:49.912897 30796 ts_tablet_manager.cc:1403] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.458s user 0.018s sys 0.015s
W20251025 20:03:49.917003 31160 log.cc:927] Time spent T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20251025 20:03:49.917236 30793 tablet_bootstrap.cc:492] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:49.911137 30673 raft_consensus.cc:3060] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.932607 30796 raft_consensus.cc:359] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.933692 30796 raft_consensus.cc:385] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:49.934065 30796 raft_consensus.cc:740] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:49.935700 30796 consensus_queue.cc:260] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:49.941111 30793 tablet_bootstrap.cc:654] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:49.945762 30796 ts_tablet_manager.cc:1434] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.032s user 0.009s sys 0.001s
I20251025 20:03:49.947352 30673 raft_consensus.cc:2468] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:49.947342 30796 tablet_bootstrap.cc:492] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:49.952488 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:49.976177 30570 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:49.978458 30570 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:49.956418 30567 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:49.980276 31012 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:49.977492 30992 consensus_queue.cc:1048] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251025 20:03:49.990680 30800 tablet_bootstrap.cc:492] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:49.991218 30800 ts_tablet_manager.cc:1403] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.485s user 0.035s sys 0.044s
I20251025 20:03:50.000778 30955 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:50.007588 31166 raft_consensus.cc:2804] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:50.004529 30955 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:50.008244 31166 raft_consensus.cc:697] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:50.013131 31171 consensus_queue.cc:1048] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20251025 20:03:50.013392 30800 raft_consensus.cc:359] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:50.018210 30800 raft_consensus.cc:385] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:50.018868 30800 raft_consensus.cc:740] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:50.019940 30800 consensus_queue.cc:260] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:50.009228 31166 consensus_queue.cc:237] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:50.023345 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:50.043180 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:50.044534 30793 tablet_bootstrap.cc:492] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:50.044662 30796 tablet_bootstrap.cc:654] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:50.045220 30793 ts_tablet_manager.cc:1403] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.128s user 0.013s sys 0.007s
I20251025 20:03:50.050700 30793 raft_consensus.cc:359] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:50.051469 30793 raft_consensus.cc:385] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:50.051769 30793 raft_consensus.cc:740] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:50.052430 30793 consensus_queue.cc:260] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:50.065019 30800 ts_tablet_manager.cc:1434] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.073s user 0.011s sys 0.001s
I20251025 20:03:50.083107 30793 ts_tablet_manager.cc:1434] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20251025 20:03:50.087312 30800 tablet_bootstrap.cc:492] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:50.072844 31073 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:50.066350 30955 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:50.174695 31012 raft_consensus.cc:2749] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:50.203440 30807 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:50.204137 30807 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:50.171830 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:50.233165 31148 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:50.234076 31148 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:50.171108 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 72276.1us
I20251025 20:03:50.174337 31073 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:50.321265 30807 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:50.369423 31167 raft_consensus.cc:2749] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:03:50.375505 30719 leader_election.cc:343] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:50.360432 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:50.389029 30570 leader_election.cc:343] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:50.391254 30796 tablet_bootstrap.cc:492] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
W20251025 20:03:50.382628 31173 log.cc:927] Time spent T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20251025 20:03:50.393041 30796 ts_tablet_manager.cc:1403] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.446s user 0.020s sys 0.008s
W20251025 20:03:50.397876 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 50497us
W20251025 20:03:50.399798 30716 leader_election.cc:343] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:50.400377 30716 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:50.541126 31148 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:50.509327 30992 raft_consensus.cc:2749] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:50.651345 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:50.556614 31073 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:50.773824 31180 log.cc:927] Time spent T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
W20251025 20:03:50.784819 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 245908us
W20251025 20:03:50.812767 30645 leader_election.cc:343] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:50.818678 31177 log.cc:927] Time spent T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.409s user 0.000s sys 0.001s
I20251025 20:03:50.783609 31175 consensus_queue.cc:1048] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.258s
I20251025 20:03:50.910812 30942 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:50.913235 30942 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.112569 30796 raft_consensus.cc:359] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.124015 30796 raft_consensus.cc:385] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:50.914523 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:51.124393 30796 raft_consensus.cc:740] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:51.109616 30800 tablet_bootstrap.cc:654] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:51.130585 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 388675us
I20251025 20:03:51.133812 30942 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.134462 30796 consensus_queue.cc:260] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:50.969357 31194 log.cc:927] Time spent T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.183s user 0.000s sys 0.000s
W20251025 20:03:51.128664 30716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.011s sys 0.012s
W20251025 20:03:51.138124 30716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.575s user 0.012s sys 0.012s
W20251025 20:03:51.138497 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 629455us
I20251025 20:03:51.108171 31197 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:51.114850 30597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 210) took 1130 ms. Trace:
I20251025 20:03:50.947907 30676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 213) took 1029 ms. Trace:
I20251025 20:03:51.100363 31193 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:51.144593 30501 catalog_manager.cc:5649] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
W20251025 20:03:51.137217 30748 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.005s sys 0.005s
W20251025 20:03:51.154388 30748 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.589s user 0.005s sys 0.006s
I20251025 20:03:51.159104 30796 ts_tablet_manager.cc:1434] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.762s user 0.016s sys 0.011s
I20251025 20:03:51.153959 31073 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:03:51.156028 31165 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.915s user 0.008s sys 0.004s
W20251025 20:03:51.159399 30796 ts_tablet_manager.cc:1467] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d: Tablet startup took 762ms
I20251025 20:03:51.143335 31197 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:51.203421 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 51981us
W20251025 20:03:51.201382 31165 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.974s user 0.009s sys 0.005s
I20251025 20:03:51.216307 30601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 220) took 1231 ms. Trace:
I20251025 20:03:51.216285 30992 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:51.173568 31073 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.219393 31197 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.221103 31206 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
W20251025 20:03:51.174799 30796 ts_tablet_manager.cc:1469] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d: Trace:
1025 20:03:40.243714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.227607 (+983893us) service_pool.cc:225] Handling call
1025 20:03:41.229563 (+ 1956us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.229689 (+ 126us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.340636 (+110947us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.347584 (+ 6948us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.348051 (+ 467us) inbound_call.cc:173] Queueing success response
1025 20:03:49.946805 (+8598754us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1025 20:03:50.397224 (+450419us) ts_tablet_manager.cc:1435] Starting tablet replica
1025 20:03:51.109549 (+712325us) tablet_replica.cc:261] Starting instrumentation
1025 20:03:51.110394 (+ 845us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":2467,"tablet-open.queue_time_us":8598784}
I20251025 20:03:51.151852 31193 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:51.211862 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 69862.4us
I20251025 20:03:51.143582 30676 rpcz_store.cc:276] 1025 20:03:49.915860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:49.918268 (+ 2408us) service_pool.cc:225] Handling call
1025 20:03:50.650496 (+732228us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:50.778218 (+127722us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:50.810044 (+ 31826us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:50.819459 (+ 9415us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:50.854788 (+ 35329us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:50.856973 (+ 2185us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:50.922626 (+ 65653us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:50.945117 (+ 22491us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4786432}
W20251025 20:03:51.231360 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 80050us
I20251025 20:03:51.240317 31205 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:51.225994 31206 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.270113 31206 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.270989 31207 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:51.270891 31206 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:51.273784 31207 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.246093 31203 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:51.227629 30992 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.288067 31215 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:51.289198 31210 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:51.289705 31190 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:51.271787 31205 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.307560 30599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 214) took 1464 ms. Trace:
I20251025 20:03:51.219376 30601 rpcz_store.cc:276] 1025 20:03:49.984514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:49.988860 (+ 4346us) service_pool.cc:225] Handling call
1025 20:03:51.155335 (+1166475us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.159964 (+ 4629us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.173856 (+ 13892us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.191314 (+ 17458us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.198364 (+ 7050us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.204291 (+ 5927us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.210050 (+ 5759us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.215685 (+ 5635us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9416576}
I20251025 20:03:51.313582 31201 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:51.313707 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:51.314338 31201 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.304919 30672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 217) took 1303 ms. Trace:
I20251025 20:03:51.323817 31211 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:51.314749 30753 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:51.334105 31191 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:51.334936 30600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 204) took 1462 ms. Trace:
I20251025 20:03:51.142892 30597 rpcz_store.cc:276] 1025 20:03:49.984447 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:49.984591 (+ 144us) service_pool.cc:225] Handling call
1025 20:03:49.985761 (+ 1170us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:49.989379 (+ 3618us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:49.989417 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:03:49.989956 (+ 539us) log.cc:844] Serialized 26 byte log entry
1025 20:03:50.070785 (+ 80829us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:03:50.070843 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:50.070932 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:03:51.113982 (+1043050us) raft_consensus.cc:1685] finished
1025 20:03:51.114286 (+ 304us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.114722 (+ 436us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3295,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4144,"spinlock_wait_cycles":1645184,"thread_start_us":80358,"threads_started":1,"wal-append.queue_time_us":373356}
I20251025 20:03:51.308039 30599 rpcz_store.cc:276] 1025 20:03:49.839645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:49.872112 (+ 32467us) service_pool.cc:225] Handling call
1025 20:03:51.240634 (+1368522us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.248228 (+ 7594us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.278039 (+ 29811us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.285043 (+ 7004us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.291159 (+ 6116us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.292900 (+ 1741us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.295682 (+ 2782us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.304081 (+ 8399us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5776256}
I20251025 20:03:51.334802 31191 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.302341 31215 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.310834 31203 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.335368 30600 rpcz_store.cc:276] 1025 20:03:49.872239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:49.874323 (+ 2084us) service_pool.cc:225] Handling call
1025 20:03:51.333881 (+1459558us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.334267 (+ 386us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.334493 (+ 226us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.334530 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.334585 (+ 55us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.334632 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.334713 (+ 81us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.334865 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6844672}
I20251025 20:03:51.355561 31201 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.304170 31210 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.348563 31073 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:51.315546 31190 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.361791 31203 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.329005 30672 rpcz_store.cc:276] 1025 20:03:50.000054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:50.007145 (+ 7091us) service_pool.cc:225] Handling call
1025 20:03:51.290383 (+1283238us) spinlock_profiling.cc:243] Waited 6.67 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.300787 (+ 10404us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.302823 (+ 2036us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.302863 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.302915 (+ 52us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.302952 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.303024 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.303173 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15335296}
I20251025 20:03:51.351509 31205 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.287622 31206 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.324517 31211 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.375730 30793 tablet_bootstrap.cc:492] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:51.368839 31207 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.384157 31201 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:51.377794 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 223) took 1017 ms. Trace:
I20251025 20:03:51.397625 31201 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.398082 30602 rpcz_store.cc:276] 1025 20:03:50.360534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:50.365305 (+ 4771us) service_pool.cc:225] Handling call
1025 20:03:51.376112 (+1010807us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea507ce 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.376290 (+ 178us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.377389 (+ 1099us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.377438 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.377486 (+ 48us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.377522 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.377586 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.377725 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9645440}
I20251025 20:03:51.407248 31167 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:51.410979 31193 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.384552 30992 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:51.408366 31190 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:51.416105 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:51.408074 31167 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.428243 31191 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:51.427387 31167 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.429363 31215 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:51.423877 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 208051us
I20251025 20:03:51.414726 31193 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:03:51.437813 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 185304us
I20251025 20:03:51.424050 31207 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:03:51.433950 30719 leader_election.cc:343] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:51.435698 31167 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:51.436831 31201 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.449630 31167 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.451381 30604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 222) took 1463 ms. Trace:
I20251025 20:03:51.451828 30800 tablet_bootstrap.cc:492] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:51.453118 30800 ts_tablet_manager.cc:1403] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 1.380s user 0.020s sys 0.051s
I20251025 20:03:51.458580 30747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 234) took 1050 ms. Trace:
I20251025 20:03:51.438248 31193 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.458950 31210 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.447178 31207 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:51.451973 31167 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.451977 30604 rpcz_store.cc:276] 1025 20:03:49.987465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:51.449962 (+1462497us) service_pool.cc:225] Handling call
1025 20:03:51.450463 (+ 501us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.450722 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.450769 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.450838 (+ 69us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:03:51.450885 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.450970 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.451138 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:51.481482 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:51.485523 30570 consensus_peers.cc:597] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 -> Peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Couldn't send request to peer 783f8fd6d80141b7912f66a954a6f92e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:51.493716 30645 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:51.479835 31206 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.557554 30793 tablet_bootstrap.cc:654] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:51.744619 30752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 149) took 1365 ms. Trace:
I20251025 20:03:51.477138 30750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 146) took 1299 ms. Trace:
I20251025 20:03:51.459105 30747 rpcz_store.cc:276] 1025 20:03:50.408410 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:50.411118 (+ 2708us) service_pool.cc:225] Handling call
1025 20:03:51.457861 (+1046743us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.458084 (+ 223us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.458249 (+ 165us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.458274 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.458307 (+ 33us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.458333 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.458378 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.458473 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7944064}
W20251025 20:03:51.992552 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 498874us
I20251025 20:03:51.664458 30796 tablet_bootstrap.cc:492] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:51.994678 30942 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:51.996737 30942 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:51.998740 30942 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:51.493216 31211 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.989254 30752 rpcz_store.cc:276] 1025 20:03:50.383314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:50.391649 (+ 8335us) service_pool.cc:225] Handling call
1025 20:03:51.743718 (+1352069us) spinlock_profiling.cc:243] Waited 6.06 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.743985 (+ 267us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.744232 (+ 247us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.744270 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.744326 (+ 56us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.744367 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.744446 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.744551 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13947264}
I20251025 20:03:51.508229 30746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 230) took 1129 ms. Trace:
I20251025 20:03:51.508469 30596 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 2
I20251025 20:03:51.826004 31208 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:51.666999 30595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 212) took 1602 ms. Trace:
I20251025 20:03:51.554349 30679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 221) took 1516 ms. Trace:
I20251025 20:03:51.664422 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:51.554807 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:51.579021 31224 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:51.556104 31207 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.507431 30675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 142) took 1460 ms. Trace:
I20251025 20:03:51.508116 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:51.754386 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:51.509559 31218 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.023725 31124 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:51.538044 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:52.031060 30955 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:51.579483 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:51.614773 31230 log.cc:927] Time spent T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
W20251025 20:03:52.000280 30570 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.192.195:46541 timed out after 1.566s (SENT) [suppressed 4 similar messages]
I20251025 20:03:51.555824 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:51.579788 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:52.021987 30992 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.021s sys 0.047s
I20251025 20:03:51.721311 31193 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:51.745536 30678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 146) took 1197 ms. Trace:
I20251025 20:03:51.739426 31210 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:51.554245 31209 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:51.556981 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:51.782832 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:51.757007 30673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 214) took 1761 ms. Trace:
I20251025 20:03:52.075731 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:51.743391 31189 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.020155 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:51.996088 31206 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:51.998713 31171 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:52.058081 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.034178 31172 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.034487 31197 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.023692 30746 rpcz_store.cc:276] 1025 20:03:50.340881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:50.344331 (+ 3450us) service_pool.cc:225] Handling call
1025 20:03:51.434680 (+1090349us) spinlock_profiling.cc:243] Waited 5.14 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.439569 (+ 4889us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.446789 (+ 7220us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.449080 (+ 2291us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.452592 (+ 3512us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.454578 (+ 1986us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.459948 (+ 5370us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.469859 (+ 9911us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11832192}
I20251025 20:03:52.034173 31216 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:51.613842 31229 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.039965 31148 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:52.085458 30800 raft_consensus.cc:359] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.120508 30800 raft_consensus.cc:385] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:52.120842 30800 raft_consensus.cc:740] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:52.086303 30942 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:52.081620 30992 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s user 0.023s sys 0.050s
I20251025 20:03:52.087354 31235 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:52.088326 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.051954 30601 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:52.042433 31213 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.046217 31124 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.085282 30744 raft_consensus.cc:2368] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2.
I20251025 20:03:51.594491 31167 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.139401 31182 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:52.139339 31167 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.089309 31202 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.079315 30751 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:52.084759 31207 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.046217 31218 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.057313 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.121520 30800 consensus_queue.cc:260] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.076726 30955 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.126963 30942 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.028056 30595 rpcz_store.cc:276] 1025 20:03:50.064097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:51.476107 (+1412010us) service_pool.cc:225] Handling call
1025 20:03:51.665907 (+189800us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.666203 (+ 296us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.666467 (+ 264us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.666512 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.666583 (+ 71us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.666635 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.666726 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.666908 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2806016}
I20251025 20:03:52.057387 31208 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.039650 30675 rpcz_store.cc:276] 1025 20:03:50.046596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:50.056460 (+ 9864us) service_pool.cc:225] Handling call
1025 20:03:51.478697 (+1422237us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.492119 (+ 13422us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.496132 (+ 4013us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.506790 (+ 10658us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.506890 (+ 100us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.506945 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.507023 (+ 78us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.507166 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4783360}
W20251025 20:03:52.138587 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 138347us
I20251025 20:03:52.100839 30600 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:52.097012 31210 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.120990 31197 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.143571 31192 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.110982 31209 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.172521 31202 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.141470 31205 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:52.117267 31206 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.056596 31224 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.139992 31182 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.146374 31049 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.084437 30678 rpcz_store.cc:276] 1025 20:03:50.547305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:50.549270 (+ 1965us) service_pool.cc:225] Handling call
1025 20:03:51.660771 (+1111501us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea507ce 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.661033 (+ 262us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.693389 (+ 32356us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.697403 (+ 4014us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.709365 (+ 11962us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.717658 (+ 8293us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.725093 (+ 7435us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.744556 (+ 19463us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8856448}
I20251025 20:03:52.147867 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:52.126530 31148 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.154484 30672 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
W20251025 20:03:52.149463 30744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 151) took 1762 ms (client timeout 1566 ms). Trace:
I20251025 20:03:52.204598 31047 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.118949 31171 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.226043 30800 ts_tablet_manager.cc:1434] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.773s user 0.002s sys 0.019s
I20251025 20:03:52.212872 30992 consensus_queue.cc:1048] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20251025 20:03:52.032002 30679 rpcz_store.cc:276] 1025 20:03:50.037519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:50.039867 (+ 2348us) service_pool.cc:225] Handling call
1025 20:03:50.040393 (+ 526us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:50.040842 (+ 449us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:50.040896 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:03:50.041429 (+ 533us) log.cc:844] Serialized 26 byte log entry
1025 20:03:51.553811 (+1512382us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:03:51.553871 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.553967 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:03:51.554014 (+ 47us) raft_consensus.cc:1685] finished
1025 20:03:51.554023 (+ 9us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.554276 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":982,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":137,"thread_start_us":270445,"threads_started":1,"wal-append.queue_time_us":325853,"wal-append.run_cpu_time_us":5036,"wal-append.run_wall_time_us":1614635}
I20251025 20:03:52.150074 31235 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:52.130530 31172 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:52.372680 30800 ts_tablet_manager.cc:1467] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957: Tablet startup took 919ms
I20251025 20:03:52.154341 31207 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.208184 31236 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.098820 30673 rpcz_store.cc:276] 1025 20:03:49.995175 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:49.996977 (+ 1802us) service_pool.cc:225] Handling call
1025 20:03:50.052735 (+ 55758us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.593308 (+1540573us) spinlock_profiling.cc:243] Waited 4.55 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea5077c 00007fd81ea5033b 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:51.628264 (+ 34956us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.656064 (+ 27800us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:51.679076 (+ 23012us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.698786 (+ 19710us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.709698 (+ 10912us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.721863 (+ 12165us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.756899 (+ 35036us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13632640}
I20251025 20:03:51.993947 30750 rpcz_store.cc:276] 1025 20:03:50.226351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:51.428463 (+1202112us) service_pool.cc:225] Handling call
1025 20:03:51.429084 (+ 621us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:51.429415 (+ 331us) raft_consensus.cc:1534] Early marking committed up to index 1
1025 20:03:51.429985 (+ 570us) log.cc:844] Serialized 14 byte log entry
1025 20:03:51.454466 (+ 24481us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:51.456257 (+ 1791us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:51.456339 (+ 82us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:51.467629 (+ 11290us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:51.475805 (+ 8176us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":536,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":5583,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":4754,"thread_start_us":9821,"threads_started":1,"wal-append.queue_time_us":47724}
W20251025 20:03:52.220106 31232 log.cc:927] Time spent T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20251025 20:03:52.120319 31189 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.224917 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.437834 30793 tablet_bootstrap.cc:492] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:52.348253 31148 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.414997 31048 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:52.268119 31211 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.245841 31202 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.219579 31049 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.214087 31205 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.346299 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.348716 31215 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:52.346899 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.243151 31167 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.136677 31229 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.426641 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.468104 31182 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:52.468989 31182 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.298785 31047 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.469623 31182 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:52.373092 30800 ts_tablet_manager.cc:1469] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957: Trace:
1025 20:03:40.393336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.355444 (+962108us) service_pool.cc:225] Handling call
1025 20:03:41.363637 (+ 8193us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.363748 (+ 111us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.555599 (+191851us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.607455 (+ 51856us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.608038 (+ 583us) inbound_call.cc:173] Queueing success response
1025 20:03:50.066203 (+8458165us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1025 20:03:51.453477 (+1387274us) ts_tablet_manager.cc:1435] Starting tablet replica
1025 20:03:51.615328 (+161851us) tablet_replica.cc:261] Starting instrumentation
1025 20:03:51.993383 (+378055us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":26447,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":12734,"tablet-open.queue_time_us":8458227}
I20251025 20:03:52.483914 30800 tablet_bootstrap.cc:492] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:52.484211 31047 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.484753 31047 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.423496 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:52.445140 30676 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:52.248736 31224 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.148159 31213 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.421592 31236 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.460259 31202 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.346736 31234 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:52.448038 30942 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.423027 31222 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
W20251025 20:03:52.442855 30568 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.762s user 0.026s sys 0.040s
W20251025 20:03:52.503877 30568 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.823s user 0.027s sys 0.040s
I20251025 20:03:52.438593 30793 ts_tablet_manager.cc:1403] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 1.063s user 0.039s sys 0.046s
I20251025 20:03:52.447031 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.154799 31216 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.227460 30748 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:03:52.475718 31167 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.499102 30942 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.502686 30955 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:52.227155 30744 rpcz_store.cc:269] 1025 20:03:50.384380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:50.396760 (+ 12380us) service_pool.cc:225] Handling call
1025 20:03:51.664223 (+1267463us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:03:52.142929 (+478706us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40091008}
I20251025 20:03:52.234175 31192 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.485246 31047 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.446668 31048 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:52.481777 31124 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.532620 30751 raft_consensus.cc:1275] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 20:03:52.516939 30942 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.534749 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:52.535081 31048 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:52.448352 31211 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.520987 30719 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:52.507417 31234 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.522249 31197 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.503149 31172 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.472700 31215 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.530678 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:52.508308 31202 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.538053 30719 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:52.556147 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.520607 31049 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:52.516238 31193 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.519079 30748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 163) took 1010 ms. Trace:
I20251025 20:03:52.528445 31208 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:52.567034 31171 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:52.555464 31048 raft_consensus.cc:2804] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:52.519521 31167 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.557658 31210 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.559145 31218 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.536581 31242 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:52.562492 31172 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:52.510083 31222 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.787081 31205 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.794246 31191 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:52.733311 31168 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.792361 31216 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.791913 31047 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.785775 31236 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.799082 31191 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.791050 31215 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:52.809752 31236 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:52.791399 31235 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717)
I20251025 20:03:52.789809 31202 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.810423 31236 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.811782 31202 consensus_queue.cc:1048] T 775d2ddde3b44056818240b2a8402e9c P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:52.812479 31236 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.788769 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.813997 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:52.814597 30716 leader_election.cc:343] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:52.815083 30716 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:52.815361 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.816200 31208 raft_consensus.cc:2749] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:52.756678 30800 tablet_bootstrap.cc:654] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:52.817642 30716 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:52.780196 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.563658 30955 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:52.820266 30676 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:52.793179 31182 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:52.781414 31181 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.822124 31191 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:52.579636 31241 log.cc:927] Time spent T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20251025 20:03:52.817533 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.840968 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.564610 31209 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:52.564373 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:52.840210 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 232) took 1370 ms. Trace:
I20251025 20:03:52.831964 30992 raft_consensus.cc:2749] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:52.793382 31211 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.763718 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:52.563896 31193 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.733861 31229 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.712827 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:52.661255 31213 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:52.658529 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:52.896579 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 48199.8us
I20251025 20:03:52.570716 31189 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.899952 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.791126 31167 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.585083 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:52.581079 31048 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.734339 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:52.899622 31216 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:52.799376 31168 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.850715 30955 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.564182 30748 rpcz_store.cc:276] 1025 20:03:51.508878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:52.111920 (+603042us) service_pool.cc:225] Handling call
1025 20:03:52.516086 (+404166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:52.830312 31234 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:52.919692 30793 raft_consensus.cc:359] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.935910 30501 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39160 (request call id 92) took 3115 ms. Trace:
I20251025 20:03:52.942365 30793 raft_consensus.cc:385] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:52.889058 30679 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:52.866135 31172 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.942760 30793 raft_consensus.cc:740] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:52.946921 30793 consensus_queue.cc:260] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.938521 31048 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:52.927515 30604 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:52.926777 31166 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:52.840828 31181 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.947450 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:52.941558 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 355996us
I20251025 20:03:52.947212 30716 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:52.944998 30955 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:52.950655 31189 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
W20251025 20:03:52.952523 30596 consensus_meta.cc:287] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.901s user 0.011s sys 0.005s
I20251025 20:03:52.960007 30716 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:52.953691 31166 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.942472 30501 rpcz_store.cc:276] 1025 20:03:49.829523 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:49.839533 (+ 10010us) service_pool.cc:225] Handling call
1025 20:03:52.925180 (+3085647us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 20:03:51.230511 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 20:03:51.231313 (+ 802us) write_op.cc:432] Acquiring schema lock in shared mode
1025 20:03:51.231359 (+ 46us) write_op.cc:435] Acquired schema lock
1025 20:03:51.231387 (+ 28us) tablet.cc:660] Decoding operations
1025 20:03:51.231879 (+ 492us) write_op.cc:620] Acquiring the partition lock for write op
1025 20:03:51.231958 (+ 79us) write_op.cc:641] Partition lock acquired for write op
1025 20:03:51.232018 (+ 60us) tablet.cc:683] Acquiring locks for 1 operations
1025 20:03:51.232308 (+ 290us) tablet.cc:699] Row locks acquired
1025 20:03:51.232341 (+ 33us) write_op.cc:260] PREPARE: finished
1025 20:03:51.232566 (+ 225us) write_op.cc:270] Start()
1025 20:03:51.232702 (+ 136us) write_op.cc:276] Timestamp: P: 1761422631232487 usec, L: 0
1025 20:03:51.232735 (+ 33us) op_driver.cc:348] REPLICATION: starting
1025 20:03:51.233657 (+ 922us) log.cc:844] Serialized 543 byte log entry
1025 20:03:51.495864 (+262207us) op_driver.cc:464] REPLICATION: finished
1025 20:03:52.094911 (+599047us) write_op.cc:301] APPLY: starting
1025 20:03:52.099157 (+ 4246us) tablet.cc:1365] starting BulkCheckPresence
1025 20:03:52.111988 (+ 12831us) tablet.cc:1368] finished BulkCheckPresence
1025 20:03:52.143666 (+ 31678us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 20:03:52.175592 (+ 31926us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 20:03:52.215626 (+ 40034us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 20:03:52.219741 (+ 4115us) write_op.cc:312] APPLY: finished
1025 20:03:52.232723 (+ 12982us) log.cc:844] Serialized 22 byte log entry
1025 20:03:52.232943 (+ 220us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:03:52.233048 (+ 105us) write_op.cc:454] Released schema lock
1025 20:03:52.233084 (+ 36us) write_op.cc:341] FINISH: Updating metrics
1025 20:03:52.233481 (+ 397us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:03:52.233526 (+ 45us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":565139,"apply.run_cpu_time_us":2785,"apply.run_wall_time_us":148485,"num_ops":1,"prepare.queue_time_us":80007,"prepare.run_cpu_time_us":6320,"prepare.run_wall_time_us":50747,"raft.queue_time_us":116253,"raft.run_cpu_time_us":8676,"raft.run_wall_time_us":356275,"replication_time_us":247086,"thread_start_us":408117,"threads_started":4,"wal-append.queue_time_us":66410}]]}
I20251025 20:03:52.963802 31189 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.965552 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:52.942024 31216 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.977445 31191 raft_consensus.cc:2804] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:52.978078 31191 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:52.978469 31191 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:52.978595 31168 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:52.972014 30716 leader_election.cc:343] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:52.846231 31242 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:52.891182 30602 rpcz_store.cc:276] 1025 20:03:51.469927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:51.494214 (+ 24287us) service_pool.cc:225] Handling call
1025 20:03:51.507991 (+ 13777us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:52.211293 (+703302us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:52.213858 (+ 2565us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:52.220243 (+ 6385us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:52.225496 (+ 5253us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:52.839849 (+614353us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b54001c1bd4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:03:52.839932 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:52.840138 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1600709376}
I20251025 20:03:52.964720 31182 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:52.980633 30793 ts_tablet_manager.cc:1434] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.476s user 0.010s sys 0.004s
W20251025 20:03:53.085693 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 113711us
W20251025 20:03:53.091295 30793 ts_tablet_manager.cc:1467] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e: Tablet startup took 586ms
I20251025 20:03:53.042586 31181 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.035043 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:52.994700 30596 raft_consensus.cc:1275] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 20:03:52.973971 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.042583 31172 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.090206 31182 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:53.077647 31234 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.103686 30796 tablet_bootstrap.cc:654] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:53.042922 31166 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.058146 30604 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:53.022698 31222 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:53.092319 30793 ts_tablet_manager.cc:1469] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e: Trace:
1025 20:03:40.413065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:40.841217 (+428152us) service_pool.cc:225] Handling call
1025 20:03:40.887874 (+ 46657us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:40.887978 (+ 104us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.373182 (+485204us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.512041 (+138859us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.512562 (+ 521us) inbound_call.cc:173] Queueing success response
1025 20:03:51.375267 (+9862705us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1025 20:03:52.504686 (+1129419us) ts_tablet_manager.cc:1435] Starting tablet replica
1025 20:03:52.744730 (+240044us) tablet_replica.cc:261] Starting instrumentation
1025 20:03:52.780461 (+ 35731us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":24916,"raft.queue_time_us":9963,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":5864,"tablet-open.queue_time_us":9862790}
I20251025 20:03:53.107564 31182 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.099084 31235 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:53.108817 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.099071 31193 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.101437 31216 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.104993 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.119104 30800 tablet_bootstrap.cc:492] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:53.119210 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.119926 30800 ts_tablet_manager.cc:1403] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.636s user 0.019s sys 0.015s
W20251025 20:03:53.120043 30752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 164) took 1396 ms (client timeout 1711 ms). Trace:
W20251025 20:03:53.120463 30752 rpcz_store.cc:269] 1025 20:03:51.723361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:52.143333 (+419972us) service_pool.cc:225] Handling call
1025 20:03:53.119071 (+975738us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:03:53.119993 (+ 922us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3358080}
I20251025 20:03:53.121876 30793 tablet_bootstrap.cc:492] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:53.122789 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.125202 31242 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.125152 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.125552 30599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 232) took 1474 ms. Trace:
I20251025 20:03:53.126027 30598 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:53.108635 30679 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:53.128371 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.129796 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.129966 30716 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:53.130173 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.131249 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.132408 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.126056 30599 rpcz_store.cc:276] 1025 20:03:51.651388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:52.915815 (+1264427us) service_pool.cc:225] Handling call
1025 20:03:52.948160 (+ 32345us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:52.953528 (+ 5368us) raft_consensus.cc:1534] Early marking committed up to index 1
1025 20:03:53.090887 (+137359us) log.cc:844] Serialized 14 byte log entry
1025 20:03:53.125160 (+ 34273us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:53.125224 (+ 64us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:53.125279 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:53.125352 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:53.125490 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":159331,"raft.queue_time_us":112,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"thread_start_us":118226,"threads_started":2}
I20251025 20:03:53.133574 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.133886 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.134140 31190 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.134397 30599 raft_consensus.cc:2468] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:53.134544 31190 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:53.135017 31190 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.135454 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.136153 30599 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:53.137174 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.138507 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.139130 30599 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:53.110870 30596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 231) took 1649 ms. Trace:
I20251025 20:03:53.145303 30719 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.108683 31234 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.146418 30719 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.148393 30719 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.150707 30678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 227) took 1093 ms. Trace:
I20251025 20:03:53.151125 30678 rpcz_store.cc:276] 1025 20:03:52.057023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:52.226938 (+169915us) service_pool.cc:225] Handling call
1025 20:03:53.143082 (+916144us) spinlock_profiling.cc:243] Waited 7.96 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:53.144798 (+ 1716us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:53.150281 (+ 5483us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:53.150319 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:53.150376 (+ 57us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:53.150410 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:53.150503 (+ 93us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:53.150634 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18306304}
I20251025 20:03:53.152561 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.153371 30748 raft_consensus.cc:2368] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2.
I20251025 20:03:53.153756 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.154353 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.133942 31048 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.113107 31235 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.135159 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.156216 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.169333 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.168505 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.162029 31213 raft_consensus.cc:2749] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.155701 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.171684 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.175169 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.175730 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.167644 31168 raft_consensus.cc:2804] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.176262 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.176781 31168 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.176971 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.177245 31168 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.177578 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.179522 31209 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.157298 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.179559 31192 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.181108 31209 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.181396 30670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 155) took 1150 ms. Trace:
I20251025 20:03:53.181833 30670 rpcz_store.cc:276] 1025 20:03:52.030490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:52.031901 (+ 1411us) service_pool.cc:225] Handling call
1025 20:03:53.169147 (+1137246us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:03:53.181292 (+ 12145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24092928}
I20251025 20:03:53.157799 30749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 243) took 1150 ms. Trace:
I20251025 20:03:53.183238 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.183665 31209 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.156375 31207 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.183910 30671 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:03:53.185994 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20251025 20:03:53.186802 30719 leader_election.cc:343] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:53.187446 30719 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.158231 31206 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.189471 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:53.191682 30719 leader_election.cc:343] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:53.192359 30719 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:03:53.192574 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 45172.3us
I20251025 20:03:53.127496 30602 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:53.194371 31175 raft_consensus.cc:2749] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:53.196691 30598 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:53.161871 30955 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.198158 30745 raft_consensus.cc:1240] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.198331 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.199028 30599 raft_consensus.cc:2468] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:53.201375 30716 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.203176 30719 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:53.139626 30567 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:53.202766 30645 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:53.157718 31235 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.171074 30600 raft_consensus.cc:2468] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:53.167447 31181 raft_consensus.cc:2804] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.207204 31215 raft_consensus.cc:2749] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.206734 31192 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:53.207408 31122 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.208760 31248 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:53.182864 30749 rpcz_store.cc:276] 1025 20:03:52.007752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:53.154907 (+1147155us) service_pool.cc:225] Handling call
1025 20:03:53.157249 (+ 2342us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:53.157443 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:53.157471 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:53.157518 (+ 47us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:03:53.157558 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:53.157618 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:53.157739 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:53.221946 31192 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.222172 31248 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.222925 31185 raft_consensus.cc:2804] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.223323 31185 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:53.223675 31185 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.223985 31192 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.224191 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.209622 31189 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.220001 31223 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:53.214625 31048 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.175966 30595 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:53.210593 30570 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.194128 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.194638 31213 raft_consensus.cc:2749] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.190521 31209 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.220214 30716 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.243453 31048 raft_consensus.cc:2749] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.243533 30716 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:53.220228 30719 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.220762 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.225539 30645 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:53.246503 30645 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:53.246815 31207 raft_consensus.cc:2804] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.247056 31248 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.247305 31207 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:53.247735 31207 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.249167 31206 raft_consensus.cc:2804] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.207283 31175 raft_consensus.cc:2804] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.249563 31206 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:53.249867 31175 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:03:53.221340 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 81802.4us
I20251025 20:03:53.250483 31175 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.250903 30567 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:53.217684 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:53.251791 30748 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:53.227488 31223 raft_consensus.cc:697] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:53.239590 31202 raft_consensus.cc:2749] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.240518 30800 raft_consensus.cc:359] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.227171 31122 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.243384 30674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 151) took 1214 ms. Trace:
I20251025 20:03:53.258147 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.259940 30719 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.224889 31182 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.262195 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.262894 31213 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:53.263451 31213 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.264137 31182 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.265288 30752 raft_consensus.cc:1240] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.265471 31213 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.262917 30747 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:53.266526 31215 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.266899 31215 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.267238 31215 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.267695 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.226729 31181 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:03:53.211215 31249 log.cc:927] Time spent T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20251025 20:03:53.256789 30800 raft_consensus.cc:385] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:53.231588 30500 catalog_manager.cc:5649] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:53.275049 30800 raft_consensus.cc:740] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:53.275791 30800 consensus_queue.cc:260] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.140208 30596 rpcz_store.cc:276] 1025 20:03:51.460932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:51.466261 (+ 5329us) service_pool.cc:225] Handling call
1025 20:03:51.488450 (+ 22189us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:53.100945 (+1612495us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:53.110566 (+ 9621us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:03:53.257917 30992 raft_consensus.cc:2749] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.252523 31048 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:53.235934 30570 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.250008 31206 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.258641 30674 rpcz_store.cc:276] 1025 20:03:52.029184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:52.031718 (+ 2534us) service_pool.cc:225] Handling call
1025 20:03:52.139438 (+107720us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:52.139632 (+ 194us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:52.390437 (+250805us) raft_consensus.cc:1534] Early marking committed up to index 1
1025 20:03:52.510685 (+120248us) log.cc:844] Serialized 14 byte log entry
1025 20:03:52.921189 (+410504us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:52.922800 (+ 1611us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:52.930226 (+ 7426us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:52.963161 (+ 32935us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b18000cf400. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8278d09a1 00007fd8278dd371 00007fd81ea8d992 00007fd81ea8fd61 00007fd81ea8fd19 00007fd81ea8fca9 00007fd81ea8fc71 00007fd81ea8ef6d 0000000000379254 0000000000379099 00007fd81ea70d1a
1025 20:03:53.242006 (+278845us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54000b6b54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:03:53.242064 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:53.243313 (+ 1249us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":258092,"prepare.queue_time_us":141260,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":262334,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":14673,"spinlock_wait_cycles":447508224,"thread_start_us":176993,"threads_started":2,"wal-append.queue_time_us":405276}
I20251025 20:03:53.268378 30745 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:53.271795 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.296404 30676 raft_consensus.cc:2468] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:53.254139 31209 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.297768 30716 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:53.298087 30567 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.299110 31234 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.299300 31209 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.300100 31209 raft_consensus.cc:2804] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.300376 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.300565 31209 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:53.301095 31209 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.278297 30500 catalog_manager.cc:5649] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 2, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 2 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:53.305088 31193 consensus_queue.cc:1048] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:53.310119 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.282215 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.283105 30800 ts_tablet_manager.cc:1434] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.163s user 0.003s sys 0.005s
I20251025 20:03:53.283532 31202 raft_consensus.cc:2749] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.311522 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.312191 30745 raft_consensus.cc:2468] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:53.312577 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.313127 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.283454 31191 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
W20251025 20:03:53.284184 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 48365.3us
I20251025 20:03:53.314934 31207 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.315307 30570 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.315629 31191 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.317054 30570 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.317183 31207 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.317857 30753 raft_consensus.cc:1240] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.319244 31166 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.319634 31166 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.320011 31166 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.284871 31048 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:53.287277 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.322642 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.287392 30719 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.323348 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.273661 31223 consensus_queue.cc:237] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.288197 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.288661 31122 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.288213 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.289268 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.292968 31168 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.272841 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.289678 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.320031 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.303988 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.274747 31181 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.279333 31242 raft_consensus.cc:2804] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.322094 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.325963 30752 raft_consensus.cc:1240] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.354574 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.327280 30570 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.324951 31048 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:53.301187 31234 raft_consensus.cc:2749] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.330370 30992 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.342499 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.329203 31248 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:03:53.325469 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.336658 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.368257 31190 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.369371 31248 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.357295 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.370757 31248 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:53.371361 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.345085 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.351490 31242 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:53.378981 30570 leader_election.cc:400] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2.
I20251025 20:03:53.362711 31122 raft_consensus.cc:2749] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.357295 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.361784 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.365545 30751 raft_consensus.cc:1240] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.372769 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.364027 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.380169 30570 leader_election.cc:403] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:03:53.364428 30992 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.385865 30719 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.381130 31242 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.394102 30672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 232) took 1272 ms. Trace:
I20251025 20:03:53.394171 31208 raft_consensus.cc:2749] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.394793 30955 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.392377 30716 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.395308 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.399161 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.399605 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.400902 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.394542 30672 rpcz_store.cc:276] 1025 20:03:52.121852 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:52.526618 (+404766us) service_pool.cc:225] Handling call
1025 20:03:52.555021 (+ 28403us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:52.865539 (+310518us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:52.872296 (+ 6757us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:03:52.941447 (+ 69151us) log.cc:844] Serialized 26 byte log entry
1025 20:03:52.994772 (+ 53325us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:03:53.008776 (+ 14004us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:53.050393 (+ 41617us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:03:53.393549 (+343156us) raft_consensus.cc:1685] finished
1025 20:03:53.393871 (+ 322us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:53.394037 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9541,"raft.queue_time_us":323654,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":23042,"thread_start_us":19596,"threads_started":1,"wal-append.queue_time_us":162892}
I20251025 20:03:53.391510 30992 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.403635 30719 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.407796 30955 raft_consensus.cc:2749] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.410286 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.410377 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.411098 30595 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.419346 30796 tablet_bootstrap.cc:492] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:53.428021 30796 ts_tablet_manager.cc:1403] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 1.764s user 0.023s sys 0.073s
I20251025 20:03:53.437070 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.420179 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.450845 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.404639 31190 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.414995 31199 raft_consensus.cc:2749] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.407328 31203 raft_consensus.cc:2749] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2.
I20251025 20:03:53.482730 30645 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:53.483278 30603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 235) took 1265 ms (client timeout 1654 ms). Trace:
I20251025 20:03:53.471539 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.426607 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.493219 30800 tablet_bootstrap.cc:492] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:53.444234 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:53.483668 30603 rpcz_store.cc:269] 1025 20:03:52.202195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:52.203290 (+ 1095us) service_pool.cc:225] Handling call
1025 20:03:53.158495 (+955205us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:03:53.214935 (+ 56440us) spinlock_profiling.cc:243] Waited 9.9 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea5077c 00007fd81ea5033b 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:03:53.340878 (+125943us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea507ce 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:03:53.450711 (+109833us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea5086c 00007fd81ea507d6 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:03:53.467077 (+ 16366us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78593280}
I20251025 20:03:53.490943 31207 consensus_queue.cc:1048] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20251025 20:03:53.482774 31168 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.511018 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:53.486052 31167 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:53.503479 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.517473 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.519531 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.520692 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.521786 31167 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.523175 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.503672 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.531502 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.531591 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.532647 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.535296 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.535923 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.537338 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.538378 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.539388 30602 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:53.549007 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 66378.5us
I20251025 20:03:53.533288 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.575677 30645 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.560860 31207 raft_consensus.cc:2749] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.563082 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.565510 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.566195 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.567282 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.568140 31167 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.568653 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.568946 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.420480 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.452082 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.507344 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.586519 30793 tablet_bootstrap.cc:654] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:53.588138 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.594848 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.598531 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.582293 31192 consensus_queue.cc:1048] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20251025 20:03:53.578536 31205 raft_consensus.cc:2749] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.605252 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.578809 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.617308 30719 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.595975 30753 raft_consensus.cc:3060] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:53.625046 31261 raft_consensus.cc:2749] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.600858 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.587809 30746 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:53.618013 30672 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.635171 30796 raft_consensus.cc:359] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.637795 30796 raft_consensus.cc:385] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:53.638254 30796 raft_consensus.cc:740] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:53.642911 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.626596 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.641065 30796 consensus_queue.cc:260] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.647563 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.625986 31048 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.633389 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.630409 31234 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.620801 31175 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.652736 30796 ts_tablet_manager.cc:1434] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.224s user 0.006s sys 0.003s
I20251025 20:03:53.635506 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.657385 30645 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:53.639060 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.656384 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.659631 30645 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:53.620826 31189 consensus_queue.cc:1059] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251025 20:03:53.665632 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.678761 31175 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.680084 30800 tablet_bootstrap.cc:654] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:53.660388 31234 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.662130 31248 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.683308 30747 raft_consensus.cc:1240] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.684214 31234 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.684970 31259 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:53.685518 31259 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.687371 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.687564 31259 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.690456 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.690192 31215 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.692891 31215 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.693960 31206 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.696058 31206 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.699373 30567 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.700515 31185 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.702343 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.702711 31185 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.703344 31185 raft_consensus.cc:2749] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:03:53.703934 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 44361.8us
W20251025 20:03:53.705158 30645 leader_election.cc:343] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:53.705592 30595 raft_consensus.cc:2468] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:53.682152 30796 tablet_bootstrap.cc:492] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:53.706684 30645 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.707595 30716 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.644673 31166 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.712162 30716 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.664407 31223 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.717268 30603 raft_consensus.cc:1240] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.710255 31222 consensus_queue.cc:1059] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:53.718681 31223 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:53.720329 31223 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.685267 31248 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.723718 30603 raft_consensus.cc:1240] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: []
I20251025 20:03:53.730525 31048 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.715555 31259 raft_consensus.cc:2749] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.733109 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.701606 30567 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.718456 31215 raft_consensus.cc:2749] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.735718 30670 raft_consensus.cc:1240] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.731724 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.744647 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.723632 31235 raft_consensus.cc:2749] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.749789 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.752856 31181 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.745088 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.754505 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.757009 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.760653 30645 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.767813 30992 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.771255 31166 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:53.771140 31209 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.772776 30672 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:03:53.784056 30645 consensus_peers.cc:597] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:53.788542 31242 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.769235 31248 raft_consensus.cc:2749] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.797652 30645 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.802795 30716 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:53.796177 31199 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:53.804605 30645 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.790885 30670 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:53.806068 30570 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.793299 31222 raft_consensus.cc:2749] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.774981 30671 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:53.812250 30570 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.817945 30752 raft_consensus.cc:2393] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
W20251025 20:03:53.790102 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 88616.3us
I20251025 20:03:53.818506 30751 raft_consensus.cc:1240] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.818506 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.819294 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:53.819553 31242 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:53.837685 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 215407us
I20251025 20:03:53.777025 30955 consensus_queue.cc:1059] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.029s
I20251025 20:03:53.847543 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.848634 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.844359 30570 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:53.809782 30679 raft_consensus.cc:1240] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.852331 30716 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:53.810508 31216 raft_consensus.cc:2804] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.853922 30603 raft_consensus.cc:1240] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.837495 31167 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.820031 31206 raft_consensus.cc:2804] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.857856 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.862061 30570 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.862217 31206 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:53.874598 31206 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.870555 30793 tablet_bootstrap.cc:492] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:53.875327 31207 raft_consensus.cc:3055] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:53.875819 31207 raft_consensus.cc:740] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:53.876749 31207 consensus_queue.cc:260] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.875833 30793 ts_tablet_manager.cc:1403] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.761s user 0.019s sys 0.015s
I20251025 20:03:53.851644 31248 raft_consensus.cc:2749] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.874430 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:53.811295 31199 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.859467 31216 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:53.869674 31209 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.884838 31215 raft_consensus.cc:2804] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.878736 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:53.873075 30567 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.861568 30601 raft_consensus.cc:2393] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:53.856026 31203 raft_consensus.cc:2804] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20251025 20:03:53.873739 31265 log.cc:927] Time spent T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
W20251025 20:03:53.860040 30748 consensus_meta.cc:287] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.608s user 0.011s sys 0.015s
I20251025 20:03:53.840476 31235 consensus_queue.cc:1048] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20251025 20:03:53.897323 30570 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.810909 31048 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:53.862170 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:53.893591 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 84892.4us
I20251025 20:03:53.871905 30992 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.885789 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.901635 30570 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:53.901870 30645 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.902840 31234 consensus_queue.cc:1048] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:03:53.903117 30645 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.892580 31215 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:53.896059 31248 raft_consensus.cc:2749] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.892023 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:53.900167 30796 tablet_bootstrap.cc:654] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:53.892238 31216 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.889740 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.864707 31193 consensus_queue.cc:1059] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20251025 20:03:53.899696 30719 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.911566 30719 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.905709 30645 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.874070 31167 consensus_queue.cc:1059] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20251025 20:03:53.902525 31122 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.900842 31185 raft_consensus.cc:2749] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:53.912827 30645 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.900017 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:53.897749 31203 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.900696 31166 raft_consensus.cc:2749] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.904656 31207 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.914013 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.903967 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:53.915081 31122 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:53.899117 30748 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:53.906237 31215 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.908520 31189 raft_consensus.cc:2749] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.904682 31274 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.941884 30719 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:53.914223 31205 raft_consensus.cc:2749] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.917951 31203 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.953837 30800 tablet_bootstrap.cc:492] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:53.954139 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:53.954466 30800 ts_tablet_manager.cc:1403] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.468s user 0.021s sys 0.020s
I20251025 20:03:53.918495 31222 raft_consensus.cc:2749] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.907891 31048 raft_consensus.cc:697] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:53.960734 31048 consensus_queue.cc:237] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.926316 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:53.926695 30751 raft_consensus.cc:1240] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3. Ops: []
I20251025 20:03:53.966187 30750 raft_consensus.cc:2393] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:53.927214 31122 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.926990 30671 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.938264 31262 raft_consensus.cc:2749] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:53.950744 31213 raft_consensus.cc:2749] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251025 20:03:53.974268 30645 leader_election.cc:343] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:53.978111 30752 raft_consensus.cc:1240] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:53.978390 30645 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:53.979466 31272 log.cc:927] Time spent T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20251025 20:03:53.980506 31199 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:53.920989 31270 raft_consensus.cc:2804] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:53.984786 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 238) took 1455 ms. Trace:
I20251025 20:03:53.987360 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:53.972666 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:03:53.989550 30719 leader_election.cc:343] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:53.950326 30567 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.990690 30645 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.950776 31268 consensus_queue.cc:1059] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20251025 20:03:53.993543 31192 raft_consensus.cc:2804] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:53.993418 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:53.998903 31268 raft_consensus.cc:3055] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:53.999274 31268 raft_consensus.cc:740] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:03:53.999370 30716 leader_election.cc:343] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:53.999816 30716 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:54.000120 31268 consensus_queue.cc:260] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.000905 31213 raft_consensus.cc:2749] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:53.997376 30793 raft_consensus.cc:359] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.004634 30793 raft_consensus.cc:385] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:54.005059 30793 raft_consensus.cc:740] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:54.004756 30645 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.004869 30800 raft_consensus.cc:359] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.006237 30800 raft_consensus.cc:385] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:54.005884 30793 consensus_queue.cc:260] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.006508 30800 raft_consensus.cc:740] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:54.007102 30800 consensus_queue.cc:260] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.003216 31268 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:53.995107 31281 raft_consensus.cc:2749] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:53.988119 31208 raft_consensus.cc:3055] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
W20251025 20:03:54.001339 31259 consensus_queue.cc:1175] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "583af532dff04e8ebba45013323d516d" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20251025 20:03:53.988381 30602 rpcz_store.cc:276] 1025 20:03:52.529650 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:53.506839 (+977189us) service_pool.cc:225] Handling call
1025 20:03:53.539131 (+ 32292us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:53.776504 (+237373us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:53.788198 (+ 11694us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:03:53.840932 (+ 52734us) log.cc:844] Serialized 26 byte log entry
1025 20:03:53.910687 (+ 69755us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:53.973337 (+ 62650us) log.cc:844] Serialized 14 byte log entry
1025 20:03:53.979198 (+ 5861us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:53.979287 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:03:53.981414 (+ 2127us) raft_consensus.cc:1685] finished
1025 20:03:53.982552 (+ 1138us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:53.984720 (+ 2168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9275,"prepare.queue_time_us":73215,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":13012,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":10006,"thread_start_us":96357,"threads_started":2,"wal-append.queue_time_us":57845}
I20251025 20:03:54.020097 30793 ts_tablet_manager.cc:1434] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.139s user 0.007s sys 0.000s
I20251025 20:03:54.008579 31235 raft_consensus.cc:2749] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.022006 31185 raft_consensus.cc:3055] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.022423 31185 raft_consensus.cc:740] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:54.023223 31185 consensus_queue.cc:260] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:54.024427 31185 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.032308 30800 ts_tablet_manager.cc:1434] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.078s user 0.006s sys 0.000s
I20251025 20:03:54.032598 30599 raft_consensus.cc:1240] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:54.032945 31223 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:53.989884 31166 consensus_queue.cc:1059] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
W20251025 20:03:54.034296 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 106442us
I20251025 20:03:54.034740 31166 raft_consensus.cc:3055] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:03:54.035115 31166 raft_consensus.cc:740] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:54.035209 30567 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.036026 31166 consensus_queue.cc:260] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.036593 30567 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.037336 31166 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:54.037505 31242 raft_consensus.cc:2749] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.036051 31167 raft_consensus.cc:2749] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.038743 30567 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.039775 31191 consensus_queue.cc:1059] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:54.041528 30992 raft_consensus.cc:3055] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.041860 30992 raft_consensus.cc:740] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:54.012904 31193 raft_consensus.cc:2749] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:03:54.043546 31203 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.046152 30645 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:53.991334 31270 raft_consensus.cc:697] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:54.048854 30645 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.051491 30645 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.012662 31208 raft_consensus.cc:740] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:54.052592 31189 raft_consensus.cc:2749] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.053201 31248 raft_consensus.cc:3055] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.053617 31248 raft_consensus.cc:740] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:54.055177 31248 consensus_queue.cc:260] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.056595 31248 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.048791 31222 raft_consensus.cc:2749] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.058602 31181 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:54.059185 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:54.060415 30793 tablet_bootstrap.cc:492] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:54.060961 31203 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:54.056526 31192 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:54.064020 30645 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.067428 31192 raft_consensus.cc:3060] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.071063 30645 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.047603 31278 raft_consensus.cc:2749] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.076637 31281 consensus_queue.cc:1059] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:03:54.063498 31277 raft_consensus.cc:2749] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.063212 31206 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.156054 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:54.158601 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:54.042676 30992 consensus_queue.cc:260] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.074563 31189 raft_consensus.cc:2749] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.078346 31235 raft_consensus.cc:2749] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.109051 30675 raft_consensus.cc:1240] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:54.077246 31275 consensus_queue.cc:1059] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:54.174247 31270 consensus_queue.cc:237] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.167115 30719 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:54.171384 30800 tablet_bootstrap.cc:492] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:54.108065 31148 raft_consensus.cc:3055] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.174499 31206 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:54.179369 31274 raft_consensus.cc:3055] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:03:54.162739 31279 consensus_queue.cc:1059] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20251025 20:03:54.177731 31199 raft_consensus.cc:2749] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.053511 31208 consensus_queue.cc:260] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.181838 30796 tablet_bootstrap.cc:492] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:54.186040 30796 ts_tablet_manager.cc:1403] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.516s user 0.046s sys 0.027s
I20251025 20:03:54.166699 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:54.176580 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:54.183097 31279 consensus_queue.cc:1243] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer 783f8fd6d80141b7912f66a954a6f92e log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251025 20:03:54.177269 31148 raft_consensus.cc:740] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:54.182730 31274 raft_consensus.cc:740] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:54.182238 31223 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:54.183400 31281 raft_consensus.cc:3055] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.163491 30567 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.191488 31235 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:54.190968 30670 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:54.207124 31148 consensus_queue.cc:260] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.208793 31279 raft_consensus.cc:2749] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.209668 31148 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.214517 31274 consensus_queue.cc:260] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.215653 30746 raft_consensus.cc:1240] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:54.216864 31274 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:54.201272 31281 raft_consensus.cc:740] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:54.218377 30796 raft_consensus.cc:359] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.221087 30796 raft_consensus.cc:385] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:54.221321 30796 raft_consensus.cc:740] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:54.222579 30796 consensus_queue.cc:260] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.208563 31235 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.210445 31222 raft_consensus.cc:2749] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251025 20:03:54.238122 30716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:03:54.220649 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 112698us
W20251025 20:03:54.246623 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:54.241611 30673 raft_consensus.cc:1240] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3. Ops: []
I20251025 20:03:54.227890 31278 consensus_queue.cc:1059] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251025 20:03:54.274044 30992 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.231806 31216 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.274679 31185 raft_consensus.cc:3055] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.275050 31185 raft_consensus.cc:740] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:54.275279 30672 raft_consensus.cc:1240] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:54.275873 31185 consensus_queue.cc:260] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.276413 31216 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:54.277170 31185 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.263717 31203 raft_consensus.cc:3055] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.263891 30716 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:54.264492 30567 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.266993 31235 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:54.270120 30670 raft_consensus.cc:1240] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:54.271720 30796 ts_tablet_manager.cc:1434] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.084s user 0.006s sys 0.002s
W20251025 20:03:54.355248 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 71951.7us
I20251025 20:03:54.356834 30716 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
W20251025 20:03:54.367908 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 80274.1us
I20251025 20:03:54.371426 30955 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:54.372110 30955 raft_consensus.cc:697] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:54.282506 31203 raft_consensus.cc:740] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:54.373050 30955 consensus_queue.cc:237] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.384956 30796 tablet_bootstrap.cc:492] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:54.354951 31259 raft_consensus.cc:2749] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251025 20:03:54.392822 30569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:54.383847 30671 raft_consensus.cc:2468] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:54.366604 31167 raft_consensus.cc:2749] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.384286 31215 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.393229 31122 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:54.409309 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 333819us
I20251025 20:03:54.362931 31283 consensus_queue.cc:1059] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20251025 20:03:54.411442 31215 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:54.416612 30793 tablet_bootstrap.cc:654] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:54.416374 30645 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.272042 31281 consensus_queue.cc:260] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.423736 30748 raft_consensus.cc:1240] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:54.418283 31122 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:54.436012 31289 log.cc:927] Time spent T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20251025 20:03:54.437650 30749 raft_consensus.cc:1240] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:54.424187 31192 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.426136 31207 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:03:54.390367 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 125950us
W20251025 20:03:54.481307 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 73109.5us
I20251025 20:03:54.487340 30500 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39172 (request call id 88) took 1647 ms. Trace:
I20251025 20:03:54.488353 30567 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.421118 31203 consensus_queue.cc:260] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.489722 30567 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.490664 31166 raft_consensus.cc:2749] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 20:03:54.518780 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:54.465512 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:54.517732 31122 raft_consensus.cc:2749] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.527505 31192 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:54.536353 31268 consensus_meta.cc:287] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.528s user 0.001s sys 0.011s
I20251025 20:03:54.542423 31191 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:54.543169 31191 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:54.566296 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 76392.9us
I20251025 20:03:54.567931 30716 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:54.569022 31190 consensus_queue.cc:1059] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:54.569437 31048 consensus_queue.cc:1059] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:54.570663 31190 raft_consensus.cc:3055] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.570961 31048 raft_consensus.cc:3055] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.571136 31190 raft_consensus.cc:740] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:54.571588 31048 raft_consensus.cc:740] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:54.572711 31048 consensus_queue.cc:260] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.574360 31268 raft_consensus.cc:2804] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:54.572331 31190 consensus_queue.cc:260] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.584447 31191 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:54.640413 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 150761us
I20251025 20:03:54.641340 30567 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.643039 30567 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.646462 30800 tablet_bootstrap.cc:654] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:54.660835 30676 raft_consensus.cc:1240] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
W20251025 20:03:54.662580 30570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:03:54.677623 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:54.678831 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:54.643599 31185 raft_consensus.cc:2749] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.654418 31208 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.653084 31203 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.661805 31122 raft_consensus.cc:2749] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.662505 31168 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:54.645951 31268 raft_consensus.cc:697] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:54.684942 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:54.686293 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:54.675190 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:54.676121 31281 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.661489 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:03:54.694248 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:54.678906 31048 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.683821 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:54.488112 30500 rpcz_store.cc:276] 1025 20:03:52.813861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:52.814086 (+ 225us) service_pool.cc:225] Handling call
1025 20:03:54.461214 (+1647128us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 20:03:53.428572 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 20:03:53.503235 (+ 74663us) write_op.cc:432] Acquiring schema lock in shared mode
1025 20:03:53.503269 (+ 34us) write_op.cc:435] Acquired schema lock
1025 20:03:53.503287 (+ 18us) tablet.cc:660] Decoding operations
1025 20:03:53.503838 (+ 551us) write_op.cc:620] Acquiring the partition lock for write op
1025 20:03:53.503927 (+ 89us) write_op.cc:641] Partition lock acquired for write op
1025 20:03:53.503954 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
1025 20:03:53.576504 (+ 72550us) tablet.cc:699] Row locks acquired
1025 20:03:53.579000 (+ 2496us) write_op.cc:260] PREPARE: finished
1025 20:03:53.587559 (+ 8559us) write_op.cc:270] Start()
1025 20:03:53.601439 (+ 13880us) write_op.cc:276] Timestamp: P: 1761422633587498 usec, L: 0
1025 20:03:53.607811 (+ 6372us) op_driver.cc:348] REPLICATION: starting
1025 20:03:53.650310 (+ 42499us) log.cc:844] Serialized 934 byte log entry
1025 20:03:54.184184 (+533874us) op_driver.cc:464] REPLICATION: finished
1025 20:03:54.236589 (+ 52405us) write_op.cc:301] APPLY: starting
1025 20:03:54.238601 (+ 2012us) tablet.cc:1365] starting BulkCheckPresence
1025 20:03:54.245736 (+ 7135us) tablet.cc:1368] finished BulkCheckPresence
1025 20:03:54.263564 (+ 17828us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 20:03:54.263966 (+ 402us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 20:03:54.264146 (+ 180us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 20:03:54.264194 (+ 48us) write_op.cc:312] APPLY: finished
1025 20:03:54.264419 (+ 225us) log.cc:844] Serialized 28 byte log entry
1025 20:03:54.264626 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:03:54.264724 (+ 98us) write_op.cc:454] Released schema lock
1025 20:03:54.264764 (+ 40us) write_op.cc:341] FINISH: Updating metrics
1025 20:03:54.265129 (+ 365us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:03:54.265160 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":96058,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":37963,"apply.run_cpu_time_us":2142,"apply.run_wall_time_us":36862,"num_ops":2,"prepare.queue_time_us":91796,"prepare.run_cpu_time_us":66869,"prepare.run_wall_time_us":317165,"raft.queue_time_us":221456,"raft.run_cpu_time_us":13212,"raft.run_wall_time_us":17636,"replication_time_us":576196,"thread_start_us":237503,"threads_started":4,"wal-append.queue_time_us":123048}]]}
W20251025 20:03:54.814065 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:54.693670 31207 raft_consensus.cc:2749] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.688848 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:54.825881 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:54.825770 30796 tablet_bootstrap.cc:654] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:54.740221 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:54.743032 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:03:54.822628 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:54.756505 31262 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:54.753327 31191 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
W20251025 20:03:54.831503 30569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:54.741531 31168 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.836474 31167 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:54.849798 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:54.854789 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:54.845520 31262 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.846432 31191 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.870744 30750 raft_consensus.cc:1240] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
W20251025 20:03:54.870695 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 196301us
I20251025 20:03:54.874593 30645 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.856297 31167 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.855880 31268 consensus_queue.cc:237] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.885438 31259 raft_consensus.cc:3055] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.885848 31167 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:54.886026 31259 raft_consensus.cc:740] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:54.886992 31259 consensus_queue.cc:260] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.878450 30645 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.889077 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:54.889338 31148 raft_consensus.cc:2749] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.889807 30670 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:03:54.893167 30645 leader_election.cc:343] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:54.893596 30645 leader_election.cc:304] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.894577 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:54.895103 30793 tablet_bootstrap.cc:492] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:54.895603 30793 ts_tablet_manager.cc:1403] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.835s user 0.033s sys 0.013s
I20251025 20:03:54.897331 30645 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.900314 30645 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.905329 30645 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.877230 31275 raft_consensus.cc:2749] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.902688 31148 raft_consensus.cc:2749] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.906816 31274 raft_consensus.cc:2749] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.906419 31222 raft_consensus.cc:2749] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251025 20:03:54.929312 31295 log.cc:927] Time spent T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20251025 20:03:54.936129 30645 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:54.911381 31207 raft_consensus.cc:2749] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.937273 31199 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:54.940227 31199 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.942479 31199 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:54.945152 31235 consensus_queue.cc:1059] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:54.908185 31192 consensus_queue.cc:1059] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20251025 20:03:54.953233 30676 raft_consensus.cc:1240] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:54.949251 31274 raft_consensus.cc:2749] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:54.961261 30676 raft_consensus.cc:1240] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:54.962325 31191 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:54.963519 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:54.954857 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:54.966006 30719 leader_election.cc:343] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:54.968036 31192 raft_consensus.cc:3055] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.968434 31192 raft_consensus.cc:740] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:54.967885 31235 consensus_queue.cc:1059] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:54.969389 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:54.969262 31192 consensus_queue.cc:260] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.969854 31207 raft_consensus.cc:3055] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:54.970453 31207 raft_consensus.cc:740] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:54.971489 31207 consensus_queue.cc:260] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.970453 31192 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:54.957985 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:54.983521 30793 raft_consensus.cc:359] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:54.987100 30793 raft_consensus.cc:385] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:54.987588 30793 raft_consensus.cc:740] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:54.966023 31168 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:54.971318 30678 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:54.959420 30677 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:03:54.992535 30796 tablet_bootstrap.cc:492] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:54.994187 30796 ts_tablet_manager.cc:1403] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.628s user 0.029s sys 0.028s
I20251025 20:03:54.988693 30793 consensus_queue.cc:260] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:54.997577 30719 leader_election.cc:343] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:54.996971 30670 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:55.011529 30672 raft_consensus.cc:1240] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:55.014617 31262 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.018041 30751 raft_consensus.cc:1240] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:55.033339 30796 raft_consensus.cc:359] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.034246 30796 raft_consensus.cc:385] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:55.034544 30796 raft_consensus.cc:740] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:55.021739 30746 raft_consensus.cc:2468] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:03:55.024880 31234 consensus_queue.cc:1059] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20251025 20:03:55.043143 30800 tablet_bootstrap.cc:492] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:55.043856 30800 ts_tablet_manager.cc:1403] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.873s user 0.024s sys 0.043s
I20251025 20:03:55.035506 30796 consensus_queue.cc:260] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.049417 30796 ts_tablet_manager.cc:1434] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.055s user 0.007s sys 0.000s
I20251025 20:03:55.024758 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.055523 30596 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:55.063587 31262 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.064244 31262 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.065119 30793 ts_tablet_manager.cc:1434] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.169s user 0.008s sys 0.000s
I20251025 20:03:55.057628 30800 raft_consensus.cc:359] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.067335 30800 raft_consensus.cc:385] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:55.067556 30800 raft_consensus.cc:740] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
W20251025 20:03:55.070399 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 113896us
I20251025 20:03:55.025568 30597 raft_consensus.cc:1275] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:55.068213 30800 consensus_queue.cc:260] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.078225 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.079917 30596 raft_consensus.cc:1275] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:55.074014 31209 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.080804 31209 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:55.081219 30645 leader_election.cc:343] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:55.082293 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.082758 31181 consensus_queue.cc:1048] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20251025 20:03:55.083969 30719 leader_election.cc:343] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:55.092442 30603 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:55.069053 31215 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:03:55.096138 30645 leader_election.cc:343] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:55.096673 30645 leader_election.cc:304] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.099275 30800 ts_tablet_manager.cc:1434] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.055s user 0.010s sys 0.003s
I20251025 20:03:55.087189 31205 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.160760 30716 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:55.103220 30567 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:55.097645 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:55.085242 31281 consensus_queue.cc:1059] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
W20251025 20:03:55.193014 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 89871.1us
I20251025 20:03:55.196785 30567 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.099880 31291 raft_consensus.cc:2749] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:55.097770 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:55.160089 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.210785 30567 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.211289 30800 tablet_bootstrap.cc:492] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:55.195991 31203 raft_consensus.cc:2749] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:55.099920 31222 consensus_queue.cc:1048] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.105s
I20251025 20:03:55.098176 31215 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.217154 31248 consensus_queue.cc:1059] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.030s
I20251025 20:03:55.200250 31122 raft_consensus.cc:2749] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:55.200158 31302 raft_consensus.cc:2804] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:55.200276 31205 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.247571 31215 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:55.257462 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 46809.8us
I20251025 20:03:55.252295 31299 raft_consensus.cc:3055] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:55.201993 31209 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.258452 30567 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.267416 30567 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:55.280670 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 183057us
I20251025 20:03:55.260489 31148 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.260030 31234 raft_consensus.cc:2749] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:55.259877 31299 raft_consensus.cc:740] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:55.260231 31302 raft_consensus.cc:697] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:55.293543 31299 consensus_queue.cc:260] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.293660 31302 consensus_queue.cc:237] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.294647 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:55.294992 31299 raft_consensus.cc:3060] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:55.295302 30750 raft_consensus.cc:2376] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1. Re-sending same reply.
I20251025 20:03:55.295351 31148 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.258436 30600 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:55.256675 31205 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.286134 31185 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.286494 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:55.288661 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:55.256640 31193 consensus_queue.cc:1059] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20251025 20:03:55.272943 31166 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:55.280074 31203 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:03:55.291115 31215 consensus_queue.cc:1048] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:55.333701 31168 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.364667 31209 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.375207 31299 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:03:55.488003 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 189229us
W20251025 20:03:55.511166 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 83734.4us
I20251025 20:03:55.512166 30719 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:55.516170 31299 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.492553 31203 raft_consensus.cc:697] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:55.483376 31166 raft_consensus.cc:697] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:55.413216 31185 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.525415 31122 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:55.535137 31281 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.554497 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.513433 31209 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:55.509728 31168 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.551622 30744 raft_consensus.cc:1240] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:55.546270 31148 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.548245 31248 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:55.566737 31122 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.581558 31262 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.582677 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:55.583226 31209 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.584530 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.585436 31166 consensus_queue.cc:237] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.589136 31325 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:55.589756 31325 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.591073 31185 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.591627 31185 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:55.591835 31325 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.592115 31185 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.594648 31185 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.595302 31185 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.595752 31185 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.596347 31290 raft_consensus.cc:3055] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:55.596755 31290 raft_consensus.cc:740] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:55.597066 31299 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:55.598206 31315 log.cc:927] Time spent T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20251025 20:03:55.558907 31175 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:55.598989 31207 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:55.599128 31175 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:55.567255 31281 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.599607 31175 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:55.601826 31209 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:55.602360 31209 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.603150 31168 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.604260 31209 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.605873 31281 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.589260 31203 consensus_queue.cc:237] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.583269 31190 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:55.607331 30595 raft_consensus.cc:1240] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:55.610297 31193 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.610755 31193 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.615795 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:55.616475 30674 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:55.618103 30800 tablet_bootstrap.cc:654] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:55.618512 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.621459 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.622174 30601 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:55.622298 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.623241 30598 raft_consensus.cc:1240] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:55.623737 30675 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:55.624605 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.626117 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.620827 31248 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.627245 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.627878 30675 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:55.628494 31248 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.629362 31248 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.615809 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:55.632941 31122 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.633980 31248 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.597584 31290 consensus_queue.cc:260] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.644512 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:55.645359 31248 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.653119 30567 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.597715 31185 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.613763 31193 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.633750 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.669273 30567 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.680409 30645 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
W20251025 20:03:55.683451 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 171405us
I20251025 20:03:55.691726 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.692216 31148 raft_consensus.cc:2804] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election won for term 2
W20251025 20:03:55.691154 30645 leader_election.cc:343] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:55.694489 30645 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.695277 31209 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.695806 31209 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.697007 31182 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:55.697489 31182 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.699324 30604 raft_consensus.cc:1275] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:55.699420 31182 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.700099 30604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 306) took 1049 ms. Trace:
I20251025 20:03:55.700460 30604 rpcz_store.cc:276] 1025 20:03:54.650395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:54.653321 (+ 2926us) service_pool.cc:225] Handling call
1025 20:03:55.659521 (+1006200us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:55.667535 (+ 8014us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:55.699797 (+ 32262us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:55.700036 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9185664}
I20251025 20:03:55.702119 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.703032 30679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 313) took 1050 ms. Trace:
I20251025 20:03:55.703367 30679 rpcz_store.cc:276] 1025 20:03:54.652912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:54.653392 (+ 480us) service_pool.cc:225] Handling call
1025 20:03:55.701996 (+1048604us) spinlock_profiling.cc:243] Waited 50.7 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea507ce 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:03:55.702965 (+ 969us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116615040}
I20251025 20:03:55.704787 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.705458 30679 raft_consensus.cc:2468] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:55.688971 31191 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.706465 31191 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.708978 31191 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:55.716032 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 46871us
I20251025 20:03:55.679121 31203 raft_consensus.cc:2749] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:55.722486 30716 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:55.725142 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.725869 30675 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:55.722724 30567 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.732410 30567 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:55.728264 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.734048 30676 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:55.736447 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.717060 31148 raft_consensus.cc:697] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:55.719719 31291 raft_consensus.cc:2749] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:55.738356 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.739928 30676 raft_consensus.cc:2468] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:55.727041 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.721241 31209 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:55.738749 30716 leader_election.cc:343] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:55.739416 31208 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:55.741254 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.719918 31329 raft_consensus.cc:2749] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:55.735975 31203 raft_consensus.cc:2804] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:55.745409 31148 consensus_queue.cc:237] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.866676 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.867421 30744 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:55.741753 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.738236 31181 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.831211 31193 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.868742 31181 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.869081 31193 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:55.869982 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.870869 31181 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.733004 31278 raft_consensus.cc:3055] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:55.871299 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.740525 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.863113 31203 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:55.871419 31278 raft_consensus.cc:740] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:55.832628 31291 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:55.740744 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.882757 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.875239 30676 raft_consensus.cc:2468] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:55.883567 30570 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:55.833145 30716 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:55.864575 31307 raft_consensus.cc:3055] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:55.883997 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.884101 30567 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:55.882618 30645 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:55.831877 31209 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.884766 30676 raft_consensus.cc:2393] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
W20251025 20:03:55.885918 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 148481us
I20251025 20:03:55.886076 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.886654 30750 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:55.886658 30716 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:55.868376 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.889027 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.889664 30604 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:55.871241 31193 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.891371 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:03:55.891758 30645 leader_election.cc:343] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:55.889781 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.892850 31190 raft_consensus.cc:2749] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:55.894089 30676 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:55.894680 30645 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.874831 31268 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.880353 31203 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:55.898593 30567 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:55.879817 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.899595 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.878578 31166 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:55.880156 31281 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:55.900251 30676 raft_consensus.cc:2468] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:03:55.901777 30716 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:55.905001 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.870038 31270 consensus_queue.cc:1048] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251025 20:03:55.907954 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.908794 30955 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.908581 30719 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:55.879030 31329 consensus_queue.cc:1059] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20251025 20:03:55.909633 30716 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:55.908566 30750 raft_consensus.cc:2468] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:55.910660 31329 consensus_queue.cc:1243] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer 783f8fd6d80141b7912f66a954a6f92e log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251025 20:03:55.909261 30955 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.892956 31318 consensus_queue.cc:1059] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20251025 20:03:55.900558 31048 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:03:55.920296 31334 log.cc:927] Time spent T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20251025 20:03:55.919982 31048 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.921010 30671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 310) took 1269 ms. Trace:
I20251025 20:03:55.921835 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.921432 30671 rpcz_store.cc:276] 1025 20:03:54.651865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:54.654637 (+ 2772us) service_pool.cc:225] Handling call
1025 20:03:55.571266 (+916629us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea507ce 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:55.920114 (+348848us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea5086c 00007fd81ea507d6 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:55.920270 (+ 156us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:55.920460 (+ 190us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:55.920489 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:55.920619 (+ 130us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:55.920669 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:55.920766 (+ 97us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:55.920946 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36611712}
I20251025 20:03:55.890358 31209 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.922680 31309 raft_consensus.cc:2804] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:55.923100 31309 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:55.923486 31309 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:55.897675 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.916298 31324 raft_consensus.cc:2749] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:55.890621 30597 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:55.908444 31193 raft_consensus.cc:2804] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:55.904870 30672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 195) took 1240 ms. Trace:
I20251025 20:03:55.899978 31185 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:55.888684 30674 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:55.893256 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:55.890340 31307 raft_consensus.cc:740] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:55.918852 30955 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:55.944136 30674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 191) took 1284 ms. Trace:
I20251025 20:03:55.944600 30674 rpcz_store.cc:276] 1025 20:03:54.659476 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:54.662379 (+ 2903us) service_pool.cc:225] Handling call
1025 20:03:55.615607 (+953228us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:03:55.944061 (+328454us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25502080}
I20251025 20:03:55.945145 31307 consensus_queue.cc:260] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.946499 31307 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:55.925491 30800 tablet_bootstrap.cc:492] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:55.916081 31327 raft_consensus.cc:2804] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:55.950212 31327 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:55.950611 31327 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:55.888082 31291 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.910590 31336 raft_consensus.cc:2749] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:55.899525 31235 raft_consensus.cc:2749] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:55.933004 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.934855 30716 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:55.963553 30716 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:55.964637 30716 leader_election.cc:304] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:55.965363 30955 raft_consensus.cc:2804] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:55.934871 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.965691 30955 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:55.965704 30716 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:55.966151 30955 raft_consensus.cc:3060] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:55.907049 30678 raft_consensus.cc:1275] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:55.906929 31166 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.969230 31215 raft_consensus.cc:2749] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:55.914935 31281 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.941962 30719 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:55.943159 30645 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:55.948030 30800 ts_tablet_manager.cc:1403] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.742s user 0.033s sys 0.048s
I20251025 20:03:55.948272 30567 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.888396 31222 consensus_queue.cc:1059] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20251025 20:03:55.931563 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.930150 31262 raft_consensus.cc:2804] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:55.933176 31193 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:55.935856 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.931660 31328 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:55.936116 30672 rpcz_store.cc:276] 1025 20:03:54.660660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:54.891271 (+230611us) service_pool.cc:225] Handling call
1025 20:03:55.011269 (+119998us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:55.011986 (+ 717us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:55.888604 (+876618us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b5400349c50. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3944d 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:03:55.900845 (+ 12241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":271317888}
I20251025 20:03:55.959899 30670 raft_consensus.cc:2393] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:55.939689 31234 raft_consensus.cc:3055] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:55.939950 31283 raft_consensus.cc:2749] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:55.906494 31268 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.969856 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.976037 31329 raft_consensus.cc:2804] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:55.979626 31199 raft_consensus.cc:2804] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:55.979763 30677 raft_consensus.cc:1275] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:55.976579 31193 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:55.975986 31235 raft_consensus.cc:2804] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:55.976300 30750 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:55.982367 30746 raft_consensus.cc:1240] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:55.978080 30716 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:55.987658 30570 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:55.947230 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:55.973606 31181 raft_consensus.cc:2804] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:55.995479 31166 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.961206 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:55.940814 31175 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:55.949724 31209 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.006675 31199 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:55.979892 31222 raft_consensus.cc:3055] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:55.974198 31270 raft_consensus.cc:2749] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:56.010366 30752 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:03:55.993589 31234 raft_consensus.cc:740] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
W20251025 20:03:56.013828 30719 leader_election.cc:343] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:56.010811 31235 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:56.007637 31281 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:55.999943 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:56.011672 31262 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:03:56.034258 30645 leader_election.cc:343] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:56.002326 31248 consensus_queue.cc:1059] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20251025 20:03:56.014029 31182 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:56.033185 31199 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.018051 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:56.032918 31122 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:56.020411 31336 raft_consensus.cc:2804] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:56.014317 31329 raft_consensus.cc:697] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:56.035296 31222 raft_consensus.cc:740] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:56.043746 31270 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:56.043668 31262 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.047695 31167 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:56.018855 31181 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:56.037213 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:56.041340 31235 raft_consensus.cc:3060] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:56.042074 30600 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:56.032488 31283 consensus_queue.cc:1048] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251025 20:03:56.049497 31324 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:56.065589 30955 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.050168 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.062639 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.004335 31328 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.054194 31182 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.064414 31234 consensus_queue.cc:260] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.055269 31325 consensus_queue.cc:1048] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20251025 20:03:56.072916 31324 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:56.110226 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 86534.7us
I20251025 20:03:56.255788 30676 raft_consensus.cc:1275] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:56.208310 30793 tablet_bootstrap.cc:492] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
W20251025 20:03:56.146816 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:56.256850 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:56.257449 30676 raft_consensus.cc:3060] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:56.074600 30955 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:56.196038 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:56.206270 30800 raft_consensus.cc:359] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.263514 30800 raft_consensus.cc:385] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:56.062273 31336 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:56.263929 30800 raft_consensus.cc:740] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:56.267154 30800 consensus_queue.cc:260] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.067750 31181 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:56.118599 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:56.071249 31291 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.072927 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:56.121641 31330 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:56.111805 31283 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:56.145608 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.255252 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:56.137653 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.217381 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.170928 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:56.066623 31167 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.156458 30753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 301) took 1508 ms. Trace:
W20251025 20:03:56.286412 31342 log.cc:927] Time spent T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20251025 20:03:56.065886 31270 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.267911 31336 raft_consensus.cc:3060] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:56.289525 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.298086 30800 ts_tablet_manager.cc:1434] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.326s user 0.007s sys 0.002s
I20251025 20:03:56.275759 31175 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.274475 31268 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.273394 31261 consensus_queue.cc:1048] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20251025 20:03:56.183475 31329 consensus_queue.cc:237] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.137945 31222 consensus_queue.cc:260] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.340833 30716 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:56.332741 30673 raft_consensus.cc:1275] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:56.331554 31344 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:56.296442 31283 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.331749 30672 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:56.309597 30753 rpcz_store.cc:276] 1025 20:03:54.647799 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:54.658116 (+ 10317us) service_pool.cc:225] Handling call
1025 20:03:55.887606 (+1229490us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea507ce 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:56.155267 (+267661us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea5086c 00007fd81ea507d6 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:56.155474 (+ 207us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:56.156014 (+ 540us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:56.156360 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":118437888}
I20251025 20:03:56.354820 30676 raft_consensus.cc:2468] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:56.333199 30670 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:56.335558 30674 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:56.356034 30716 leader_election.cc:304] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:56.356568 30800 tablet_bootstrap.cc:492] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:56.342880 31190 raft_consensus.cc:2749] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:56.363011 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.340260 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.366907 30796 tablet_bootstrap.cc:492] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
W20251025 20:03:56.341089 30747 consensus_meta.cc:287] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.629s user 0.007s sys 0.008s
I20251025 20:03:56.374539 30716 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:56.378363 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:56.333643 31345 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:56.381042 30676 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.372617 31261 raft_consensus.cc:2804] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
W20251025 20:03:56.384790 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 303041us
I20251025 20:03:56.385119 30570 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:56.370153 30679 raft_consensus.cc:2393] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:56.387449 30716 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:03:56.370275 31208 consensus_meta.cc:287] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.509s user 0.002s sys 0.010s
W20251025 20:03:56.389429 30645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:56.370558 31328 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.394083 31208 raft_consensus.cc:2749] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:03:56.373435 31048 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.384388 31268 raft_consensus.cc:2804] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:56.368073 31327 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.406504 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.358206 31344 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.407773 31166 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:03:56.411422 30716 leader_election.cc:343] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:56.380722 30747 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:03:56.389966 31248 raft_consensus.cc:2749] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:56.392587 31261 raft_consensus.cc:697] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:56.417389 30645 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:56.394254 31175 raft_consensus.cc:2804] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:56.425482 30679 raft_consensus.cc:1275] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:03:56.426323 31192 raft_consensus.cc:2749] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:56.412638 31268 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:56.421404 31316 consensus_queue.cc:1048] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20251025 20:03:56.424196 31166 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.419222 31182 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:56.424983 31309 consensus_meta.cc:287] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.501s user 0.006s sys 0.008s
I20251025 20:03:56.435206 31268 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.384523 31345 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.458894 31248 consensus_queue.cc:1048] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:03:56.459892 31327 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.435596 30746 raft_consensus.cc:1275] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:03:56.461402 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 297) took 1371 ms. Trace:
I20251025 20:03:56.423066 30672 raft_consensus.cc:2468] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:56.461942 30603 rpcz_store.cc:276] 1025 20:03:55.089766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:55.091866 (+ 2100us) service_pool.cc:225] Handling call
1025 20:03:55.092249 (+ 383us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:55.253774 (+161525us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:55.253814 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:03:55.254494 (+ 680us) log.cc:844] Serialized 26 byte log entry
1025 20:03:55.682161 (+427667us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:03:55.682232 (+ 71us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:55.682525 (+ 293us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:03:56.426625 (+744100us) spinlock_profiling.cc:243] Waited 9.08 ms on lock 0x7b4400026230. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e9640cc 00007fd81e964527 00007fd821702045 00007fd821702779 00007fd82170297a 00007fd821702b4d 00007fd8216da438 00007fd8216d581a 00007fd8216d4c66 00007fd8216d4888
1025 20:03:56.461083 (+ 34458us) raft_consensus.cc:1685] finished
1025 20:03:56.461108 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:56.461334 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":83,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":20874496,"thread_start_us":427280,"threads_started":1,"wal-append.queue_time_us":483757}
I20251025 20:03:56.463595 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.463929 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:56.464785 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 465708us
I20251025 20:03:56.435441 30604 raft_consensus.cc:1240] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
I20251025 20:03:56.448611 31270 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.445744 31261 consensus_queue.cc:237] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.430951 31175 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:56.466784 31175 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.456856 31167 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.461337 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.472260 31283 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.425522 31324 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.477741 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.478698 30679 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:56.480110 30679 raft_consensus.cc:1240] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
W20251025 20:03:56.514183 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20251025 20:03:56.515095 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 97813.2us
I20251025 20:03:56.518440 31166 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.464847 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.468618 31309 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:56.543299 31307 consensus_meta.cc:287] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.596s user 0.010s sys 0.008s
I20251025 20:03:56.543437 31309 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.590442 31344 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:56.598405 31203 consensus_meta.cc:287] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.687s user 0.008s sys 0.012s
I20251025 20:03:56.660260 30676 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:56.640921 31345 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:56.597757 31199 consensus_meta.cc:287] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.539s user 0.009s sys 0.000s
I20251025 20:03:56.680977 30793 tablet_bootstrap.cc:654] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:56.682469 31193 consensus_meta.cc:287] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.667s user 0.018s sys 0.002s
W20251025 20:03:56.682693 31262 consensus_meta.cc:287] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.614s user 0.023s sys 0.016s
I20251025 20:03:56.683432 31181 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.685576 31175 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:56.671216 30596 consensus_meta.cc:287] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 1.026s user 0.018s sys 0.018s
I20251025 20:03:56.670336 31203 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.692132 31193 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.694463 31344 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:56.692752 31262 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.614537 30672 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:56.716120 31175 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.718703 31349 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:56.719472 31268 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.720405 30796 tablet_bootstrap.cc:654] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:56.722234 31185 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:03:56.719404 31122 consensus_meta.cc:287] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.656s user 0.009s sys 0.013s
I20251025 20:03:56.732779 30676 raft_consensus.cc:1275] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:56.734460 31167 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:56.732422 31344 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.737949 31181 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.747622 30603 raft_consensus.cc:1240] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251025 20:03:56.723766 31193 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.738696 31349 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.754237 31167 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.721616 30596 raft_consensus.cc:2468] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:03:56.754899 31290 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:56.692167 31199 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:56.741398 31185 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:56.731930 31309 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:56.779107 30793 tablet_bootstrap.cc:492] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:56.786149 30793 ts_tablet_manager.cc:1403] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.603s user 0.019s sys 0.013s
I20251025 20:03:56.748947 31248 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:56.747601 31122 raft_consensus.cc:2749] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:56.780483 31327 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:56.780164 31181 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:56.814667 30745 raft_consensus.cc:2393] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:56.815433 31203 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.815613 31199 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.787257 31259 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:56.815697 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:03:56.787310 30596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 291) took 1701 ms (client timeout 1612 ms). Trace:
I20251025 20:03:56.818194 31181 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:56.819701 30596 rpcz_store.cc:269] 1025 20:03:55.085340 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:55.087389 (+ 2049us) service_pool.cc:225] Handling call
1025 20:03:55.644334 (+556945us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:03:56.785309 (+1140975us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2856192}
I20251025 20:03:56.820569 31268 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.822151 30800 tablet_bootstrap.cc:654] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:56.822592 30602 raft_consensus.cc:1240] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:56.824141 30645 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:56.822880 30596 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:03:56.819504 31235 consensus_meta.cc:287] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.745s user 0.007s sys 0.014s
I20251025 20:03:56.787717 31309 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:56.819682 31327 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.834159 31316 consensus_queue.cc:1059] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:56.746276 31336 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.820248 31274 raft_consensus.cc:3055] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:56.823487 30595 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.828223 31205 raft_consensus.cc:2804] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:03:56.794277 31248 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.842085 31344 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.840644 31274 raft_consensus.cc:740] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:56.836004 31235 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.848609 31349 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.852396 31274 consensus_queue.cc:260] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.854175 31327 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.855791 31336 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.857367 31309 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717)
I20251025 20:03:56.847446 31205 raft_consensus.cc:697] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:56.859205 31205 consensus_queue.cc:237] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.860716 31235 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.860576 30793 raft_consensus.cc:359] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.861052 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:56.861340 30793 raft_consensus.cc:385] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:56.861614 30793 raft_consensus.cc:740] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:56.863420 30670 raft_consensus.cc:1275] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:56.865080 31330 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:56.865151 30595 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:03:56.881422 31181 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.881938 30796 tablet_bootstrap.cc:492] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:56.882489 30796 ts_tablet_manager.cc:1403] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.516s user 0.024s sys 0.012s
I20251025 20:03:56.865767 31330 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.869203 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:56.869356 31248 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.870167 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:56.870661 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:56.857400 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:56.874161 31274 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:56.877478 30677 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:56.878329 30793 consensus_queue.cc:260] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.884285 30597 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
W20251025 20:03:56.885213 30719 leader_election.cc:343] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:56.890894 30719 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:56.892040 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:56.892556 30597 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.894510 30796 raft_consensus.cc:359] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.895151 30796 raft_consensus.cc:385] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:56.895435 30796 raft_consensus.cc:740] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:56.896585 30796 consensus_queue.cc:260] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.899120 30796 ts_tablet_manager.cc:1434] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20251025 20:03:56.889768 31307 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:56.899868 31307 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.900184 31262 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:56.901170 30671 raft_consensus.cc:2393] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:56.902763 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:56.903312 31343 log.cc:927] Time spent T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.758s user 0.001s sys 0.012s
I20251025 20:03:56.903481 30800 tablet_bootstrap.cc:492] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:56.903635 30748 raft_consensus.cc:2393] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:56.904023 30800 ts_tablet_manager.cc:1403] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.553s user 0.009s sys 0.026s
I20251025 20:03:56.904548 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:56.905030 30752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 316) took 1041 ms. Trace:
I20251025 20:03:56.905154 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:56.905501 30748 raft_consensus.cc:2393] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:56.907351 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:56.908149 30753 raft_consensus.cc:2393] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:56.905417 30752 rpcz_store.cc:276] 1025 20:03:55.863534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:55.863715 (+ 181us) service_pool.cc:225] Handling call
1025 20:03:56.904872 (+1041157us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b5400287b54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:03:56.904974 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291626880}
I20251025 20:03:56.909767 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:56.910010 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:56.911041 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:56.911296 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:56.911551 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 327) took 1223 ms. Trace:
I20251025 20:03:56.911063 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:56.906289 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:56.901476 30597 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:56.911654 30601 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.918941 30595 raft_consensus.cc:2393] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:56.890311 31185 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:56.917519 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:56.922359 30796 tablet_bootstrap.cc:492] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:56.923282 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:56.923763 30955 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:56.924957 30955 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.923986 30800 raft_consensus.cc:359] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.926262 30800 raft_consensus.cc:385] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:56.928387 30955 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:56.930043 30719 leader_election.cc:343] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:56.931001 30719 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:56.928387 30800 raft_consensus.cc:740] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:56.932410 30955 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:56.933113 30955 raft_consensus.cc:697] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:56.933244 30800 consensus_queue.cc:260] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.935215 30597 raft_consensus.cc:2393] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:56.935592 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:56.936667 30752 raft_consensus.cc:2393] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:56.917752 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:56.924295 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:56.922703 31190 raft_consensus.cc:2749] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:56.941182 30955 consensus_queue.cc:237] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.944962 30793 ts_tablet_manager.cc:1434] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.159s user 0.007s sys 0.001s
I20251025 20:03:56.946913 30645 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:56.947957 30800 ts_tablet_manager.cc:1434] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.044s user 0.001s sys 0.008s
I20251025 20:03:56.911983 30598 rpcz_store.cc:276] 1025 20:03:55.688574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:55.689966 (+ 1392us) service_pool.cc:225] Handling call
1025 20:03:55.694696 (+ 4730us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:55.723259 (+ 28563us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:55.725018 (+ 1759us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:03:55.725591 (+ 573us) log.cc:844] Serialized 26 byte log entry
1025 20:03:56.062937 (+337346us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:56.067956 (+ 5019us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:56.102017 (+ 34061us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:03:56.902774 (+800757us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:03:56.908680 (+ 5906us) raft_consensus.cc:1685] finished
1025 20:03:56.911258 (+ 2578us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:56.911488 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22236,"raft.queue_time_us":25526,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":7287,"spinlock_wait_cycles":36486912,"thread_start_us":328507,"threads_started":1,"wal-append.queue_time_us":400824}
I20251025 20:03:56.949662 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:56.949211 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:56.946592 30753 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.951684 30800 tablet_bootstrap.cc:492] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:56.951560 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:56.952668 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:56.953075 30752 raft_consensus.cc:2393] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:56.955396 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:56.956758 30597 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.948848 31309 raft_consensus.cc:2749] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:56.957602 30793 tablet_bootstrap.cc:492] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:56.964231 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:56.962486 31357 log.cc:927] Time spent T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20251025 20:03:56.963208 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:56.969693 30645 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:56.970719 30796 tablet_bootstrap.cc:654] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:56.970101 30749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 319) took 1105 ms. Trace:
I20251025 20:03:56.962530 30675 raft_consensus.cc:1240] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:56.971271 31235 raft_consensus.cc:2749] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:03:56.972615 30597 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:56.984594 30675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 217) took 1252 ms. Trace:
I20251025 20:03:56.985255 30675 rpcz_store.cc:276] 1025 20:03:55.732449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:56.033341 (+300892us) service_pool.cc:225] Handling call
1025 20:03:56.962064 (+928723us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:56.962302 (+ 238us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:56.978809 (+ 16507us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:56.984505 (+ 5696us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4930816}
I20251025 20:03:56.986419 31185 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:56.964972 30595 raft_consensus.cc:2393] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:56.991176 30719 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:56.992889 30748 raft_consensus.cc:3060] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:56.995606 30601 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:56.979171 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:56.996922 30598 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:56.998093 31209 raft_consensus.cc:2804] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:56.998826 31209 raft_consensus.cc:697] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:57.000036 31209 consensus_queue.cc:237] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.002014 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:57.002910 30595 raft_consensus.cc:2393] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:57.003494 31309 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:57.004048 31309 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:56.974411 30749 rpcz_store.cc:276] 1025 20:03:55.873543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:56.157098 (+283555us) service_pool.cc:225] Handling call
1025 20:03:56.179257 (+ 22159us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:56.280241 (+100984us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:56.282078 (+ 1837us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:56.284015 (+ 1937us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:56.284056 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:56.969812 (+685756us) spinlock_profiling.cc:243] Waited 98.5 ms on lock 0x7b54002d7154. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:03:56.969869 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:56.970036 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":226466176}
I20251025 20:03:57.003832 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:56.974205 31278 consensus_queue.cc:260] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.006155 31309 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.007176 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:57.007825 30595 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:57.007927 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:57.008721 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:03:57.011023 30747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 308) took 2359 ms (client timeout 1661 ms). Trace:
I20251025 20:03:57.011571 30600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 344) took 1129 ms. Trace:
I20251025 20:03:57.011055 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:57.012034 30600 rpcz_store.cc:276] 1025 20:03:55.881777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:56.354320 (+472543us) service_pool.cc:225] Handling call
1025 20:03:56.414871 (+ 60551us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:56.442794 (+ 27923us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:56.447502 (+ 4708us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:03:56.457332 (+ 9830us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:56.466419 (+ 9087us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:57.011314 (+544895us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b540038ed54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:03:57.011365 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:57.011514 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28597120}
I20251025 20:03:57.012903 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:57.017095 31307 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.021775 30719 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:57.023111 30955 raft_consensus.cc:2804] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:57.027770 30955 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:57.028380 30955 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:03:57.024633 30719 leader_election.cc:343] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:57.045408 31324 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:57.059772 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 345) took 1030 ms. Trace:
I20251025 20:03:57.057520 30719 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:57.064316 30604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 340) took 1196 ms. Trace:
I20251025 20:03:57.064848 30604 rpcz_store.cc:276] 1025 20:03:55.867735 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:56.287223 (+419488us) service_pool.cc:225] Handling call
1025 20:03:56.287496 (+ 273us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:56.466180 (+178684us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:57.064023 (+597843us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400121bd4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:03:57.064242 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":312520448}
W20251025 20:03:57.067790 31360 log.cc:927] Time spent T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20251025 20:03:57.067879 30750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 230) took 1200 ms. Trace:
I20251025 20:03:57.068502 30750 rpcz_store.cc:276] 1025 20:03:55.867190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:55.913391 (+ 46201us) service_pool.cc:225] Handling call
1025 20:03:57.067601 (+1154210us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b540036f954. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:03:57.067792 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23684352}
I20251025 20:03:57.056990 30599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 303) took 1525 ms. Trace:
I20251025 20:03:57.076010 30570 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:03:57.071620 30745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 234) took 1148 ms. Trace:
I20251025 20:03:57.074147 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:57.081530 30955 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.079202 31190 raft_consensus.cc:2804] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:57.070055 30677 rpcz_store.cc:276] 1025 20:03:56.029379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:56.044205 (+ 14826us) service_pool.cc:225] Handling call
1025 20:03:57.059589 (+1015384us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b54000b6154. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:03:57.059703 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1412874624}
I20251025 20:03:57.078657 30749 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:03:57.077222 31324 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.090610 30645 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:57.091631 31274 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:57.050952 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:57.092150 31274 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:57.103027 30744 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:57.103307 31274 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:57.085798 31330 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.120476 30597 raft_consensus.cc:1240] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
I20251025 20:03:57.070539 30599 rpcz_store.cc:276] 1025 20:03:55.526340 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:55.527840 (+ 1500us) service_pool.cc:225] Handling call
1025 20:03:55.725610 (+197770us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:03:55.725840 (+ 230us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:55.726293 (+ 453us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:55.726322 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:03:55.726817 (+ 495us) log.cc:844] Serialized 26 byte log entry
1025 20:03:55.886973 (+160156us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:03:55.891366 (+ 4393us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:55.896247 (+ 4881us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:03:57.024921 (+1128674us) spinlock_profiling.cc:243] Waited 4.52 ms on lock 0x7b4400026230. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e9640cc 00007fd81e964527 00007fd821702045 00007fd821702779 00007fd82170297a 00007fd821702b4d 00007fd8216da438 00007fd8216d581a 00007fd8216d4c66 00007fd8216d4888
1025 20:03:57.047103 (+ 22182us) raft_consensus.cc:1685] finished
1025 20:03:57.047139 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:03:57.051619 (+ 4480us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":803,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":5638,"spinlock_wait_cycles":19923712,"thread_start_us":151781,"threads_started":1,"wal-append.queue_time_us":162450}
W20251025 20:03:57.167564 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 89353.1us
I20251025 20:03:57.073727 31185 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:03:57.011596 30747 rpcz_store.cc:269] 1025 20:03:54.651721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:54.654478 (+ 2757us) service_pool.cc:225] Handling call
1025 20:03:57.010788 (+2356310us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b54001e6254. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:03:57.010904 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2060282112}
I20251025 20:03:57.179028 30955 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:57.169636 30719 consensus_peers.cc:597] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:57.078320 30745 rpcz_store.cc:276] 1025 20:03:55.916954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:55.929564 (+ 12610us) service_pool.cc:225] Handling call
1025 20:03:57.064589 (+1135025us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x7b5400209c54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:03:57.064805 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1775616256}
I20251025 20:03:57.079511 31242 raft_consensus.cc:2764] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251025 20:03:57.199676 30793 tablet_bootstrap.cc:654] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:57.184911 31316 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:03:57.239291 30570 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.192.195:46541 timed out after 1.578s (SENT) [suppressed 3 similar messages]
I20251025 20:03:57.185350 30753 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:03:57.195042 31325 consensus_queue.cc:1059] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251025 20:03:57.186324 31185 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.215373 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:57.217422 30596 raft_consensus.cc:1275] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:57.266624 30570 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:57.268560 30645 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:57.206324 31316 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.277118 30796 tablet_bootstrap.cc:492] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:57.282639 30796 ts_tablet_manager.cc:1403] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.383s user 0.039s sys 0.014s
I20251025 20:03:57.270380 31242 raft_consensus.cc:2749] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:03:57.273221 30595 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 3
I20251025 20:03:57.272760 31281 raft_consensus.cc:2804] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:57.291687 31190 raft_consensus.cc:697] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:57.320516 31190 consensus_queue.cc:237] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:57.321188 31167 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.334678 30800 tablet_bootstrap.cc:654] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:57.337592 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:57.312778 31166 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:57.323755 31281 raft_consensus.cc:697] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:57.354746 30750 raft_consensus.cc:1275] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:57.356519 30796 raft_consensus.cc:359] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.357481 30796 raft_consensus.cc:385] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:57.357864 30796 raft_consensus.cc:740] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:57.336182 31259 consensus_queue.cc:1059] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20251025 20:03:57.351200 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:57.362187 31259 raft_consensus.cc:3055] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:03:57.362754 31259 raft_consensus.cc:740] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:57.362181 31281 consensus_queue.cc:237] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.364053 30598 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:57.360064 30796 consensus_queue.cc:260] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:57.367954 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 45369.1us
I20251025 20:03:57.362257 30747 raft_consensus.cc:2468] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:57.366827 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:57.395547 31330 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.396158 31330 raft_consensus.cc:493] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:57.396606 31330 raft_consensus.cc:515] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.381414 30567 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:57.381732 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:57.382184 30716 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:57.400198 30796 ts_tablet_manager.cc:1434] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.117s user 0.003s sys 0.005s
I20251025 20:03:57.382295 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:57.377132 31299 consensus_queue.cc:1048] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20251025 20:03:57.403239 30645 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:57.386664 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 350) took 1408 ms. Trace:
I20251025 20:03:57.404546 30603 rpcz_store.cc:276] 1025 20:03:55.978251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:56.468426 (+490175us) service_pool.cc:225] Handling call
1025 20:03:56.468708 (+ 282us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:56.748226 (+279518us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:57.386374 (+638148us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b540012f2d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:03:57.386583 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":627981696}
I20251025 20:03:57.387727 30746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 238) took 1055 ms. Trace:
I20251025 20:03:57.391182 31316 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.394265 31185 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.394373 31336 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:57.363869 31259 consensus_queue.cc:260] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.398161 30570 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:57.350981 31166 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.401605 30716 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:57.396971 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:57.409993 30645 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:57.399230 31328 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:57.402642 31242 raft_consensus.cc:2749] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:57.402613 31262 raft_consensus.cc:2804] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251025 20:03:57.424029 30645 leader_election.cc:343] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:57.411353 31235 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:57.424598 30645 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:57.424685 30716 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:57.432305 30570 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:57.426612 30796 tablet_bootstrap.cc:492] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:57.415211 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:57.422010 31316 raft_consensus.cc:2749] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:57.449247 30716 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:57.425280 31262 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:57.429090 31325 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:57.454581 30716 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
W20251025 20:03:57.456050 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 55429.2us
I20251025 20:03:57.445515 31259 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:57.422282 31336 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.418747 31274 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.449736 31242 raft_consensus.cc:2804] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:57.465174 31242 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:57.465598 31242 raft_consensus.cc:3060] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:57.448637 31207 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:57.445091 31190 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:03:57.469189 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 45204.2us
I20251025 20:03:57.454073 31262 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:57.450671 31209 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:57.463512 31299 consensus_queue.cc:1059] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:57.464702 31274 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.443346 31368 consensus_queue.cc:1048] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20251025 20:03:57.412070 30746 rpcz_store.cc:276] 1025 20:03:56.339061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:56.343844 (+ 4783us) service_pool.cc:225] Handling call
1025 20:03:56.394465 (+ 50621us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:03:56.460718 (+ 66253us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:57.387413 (+926695us) spinlock_profiling.cc:243] Waited 9.51 ms on lock 0x7b540022fe54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:03:57.387604 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21866880}
I20251025 20:03:57.459308 31181 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:57.426944 31328 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.471680 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:57.473664 30645 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:57.475147 30955 raft_consensus.cc:3055] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:03:57.428194 31235 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.455592 31268 raft_consensus.cc:2749] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:03:57.459638 31316 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:57.467308 31248 consensus_queue.cc:1048] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251025 20:03:57.535346 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 103183us
I20251025 20:03:57.532716 31377 raft_consensus.cc:2749] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:03:57.450964 31191 consensus_queue.cc:1048] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20251025 20:03:57.543543 30955 raft_consensus.cc:740] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:57.531628 31209 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:57.460379 31378 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:57.548288 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.548377 30793 tablet_bootstrap.cc:492] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:57.544899 31222 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:03:57.557042 30570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:03:57.458365 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:03:57.557906 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 84342.3us
I20251025 20:03:57.470536 31167 consensus_queue.cc:1059] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20251025 20:03:57.539842 31181 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:57.561147 30955 consensus_queue.cc:260] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.576068 30955 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:57.534528 31190 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.533563 31379 raft_consensus.cc:2804] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:57.482630 31207 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.566514 30793 ts_tablet_manager.cc:1403] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.612s user 0.015s sys 0.042s
I20251025 20:03:57.557235 31382 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:57.589512 31382 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.558622 31316 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:57.586495 31375 log.cc:927] Time spent T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
W20251025 20:03:57.591405 31371 log.cc:927] Time spent T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.142s user 0.001s sys 0.000s
I20251025 20:03:57.586625 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.587204 31166 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.594538 31330 leader_election.cc:290] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.567083 31299 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:57.595377 31299 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.579519 31380 raft_consensus.cc:2749] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:03:57.584969 31181 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:57.564847 31209 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:57.588662 30793 raft_consensus.cc:359] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.603572 30793 raft_consensus.cc:385] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:57.603988 30793 raft_consensus.cc:740] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:57.606998 30793 consensus_queue.cc:260] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.576248 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.631863 31328 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.616470 31190 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:57.618093 30749 raft_consensus.cc:1240] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3. Ops: []
I20251025 20:03:57.633368 30674 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:57.616868 31324 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:57.634164 31207 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.619971 31374 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:57.584506 31379 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:57.635049 31374 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.635522 31379 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:57.636188 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.620263 30800 tablet_bootstrap.cc:492] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:57.619037 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:57.627648 31235 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.629873 31336 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:57.630894 31234 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:57.632118 30671 raft_consensus.cc:2368] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3.
I20251025 20:03:57.619514 31384 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:57.637444 30800 ts_tablet_manager.cc:1403] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.686s user 0.011s sys 0.023s
I20251025 20:03:57.620417 31377 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:57.647017 30793 ts_tablet_manager.cc:1434] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.065s user 0.005s sys 0.002s
I20251025 20:03:57.657439 31299 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.658766 31382 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.658977 30800 raft_consensus.cc:359] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.659860 30800 raft_consensus.cc:385] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:57.660223 30800 raft_consensus.cc:740] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:57.662634 31316 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.661046 30800 consensus_queue.cc:260] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.650691 31374 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:57.671996 30744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 228) took 1805 ms (client timeout 1578 ms). Trace:
I20251025 20:03:57.673197 30753 raft_consensus.cc:1240] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:57.675477 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:57.649577 31377 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.680835 31377 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.649732 31384 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.686782 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.680627 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.705651 30676 raft_consensus.cc:1275] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251025 20:03:57.672684 30744 rpcz_store.cc:269] 1025 20:03:55.866590 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:55.868107 (+ 1517us) service_pool.cc:225] Handling call
1025 20:03:57.671690 (+1803583us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400239454. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:03:57.671887 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2762811264}
I20251025 20:03:57.716531 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.716882 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:57.729832 31384 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.732015 30679 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:57.692667 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.733937 30674 raft_consensus.cc:2468] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:57.677774 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.738708 30677 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:57.740574 30796 tablet_bootstrap.cc:654] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:57.741071 30673 raft_consensus.cc:2468] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:57.643349 31242 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.737087 31209 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.759159 30746 raft_consensus.cc:1240] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251025 20:03:57.760073 31325 raft_consensus.cc:2749] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:57.761247 31327 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:57.761978 31327 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.764199 31262 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.765435 31379 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:03:57.767379 31327 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.758020 31378 raft_consensus.cc:2749] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:57.765859 30752 raft_consensus.cc:2368] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3.
I20251025 20:03:57.752420 31181 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.742242 31278 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:57.772300 31181 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:57.767421 31379 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717)
I20251025 20:03:57.764227 31209 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:57.779572 30793 tablet_bootstrap.cc:492] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:57.782160 30800 ts_tablet_manager.cc:1434] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.136s user 0.008s sys 0.000s
I20251025 20:03:57.783777 30800 tablet_bootstrap.cc:492] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:57.786725 31209 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:57.788834 31181 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:57.789721 31181 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.787616 31209 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.745592 31242 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.800206 31209 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:57.801548 31181 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:57.800208 31262 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:57.809842 30800 tablet_bootstrap.cc:654] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:57.815534 30793 tablet_bootstrap.cc:654] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:57.854653 30602 raft_consensus.cc:1275] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:57.860155 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.862751 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:57.865298 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.866434 30672 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:57.867995 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.902798 30751 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:03:57.911674 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:57.913074 30670 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:57.926767 31278 consensus_queue.cc:1048] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:03:57.927922 31192 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:57.928916 31379 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:57.929029 31192 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.930022 31379 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.931000 30719 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:57.932497 31192 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.932798 31379 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.921286 30677 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:57.938622 31262 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:57.939257 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:57.939585 31262 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.940878 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:57.943714 31325 raft_consensus.cc:2804] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:57.944466 31325 raft_consensus.cc:697] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:57.944285 31327 consensus_queue.cc:1059] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
W20251025 20:03:57.951613 30595 consensus_meta.cc:287] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.619s user 0.014s sys 0.001s
I20251025 20:03:57.952299 30595 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:57.941658 30677 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:57.960315 31278 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:57.963454 31278 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.965243 31325 consensus_queue.cc:237] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.971170 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.972501 30679 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:57.971488 31262 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:57.972996 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 348207us
I20251025 20:03:57.977782 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.983446 31384 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:57.984519 31384 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.986840 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:57.988368 31318 consensus_queue.cc:1048] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:57.988955 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:57.991021 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:57.991961 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:57.987643 31384 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:57.992079 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:57.997546 30752 raft_consensus.cc:1275] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:57.991082 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.003340 30672 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:58.000665 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:58.005326 30746 raft_consensus.cc:2393] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:57.992090 30679 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:57.995990 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:58.012930 30673 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.013408 31234 consensus_queue.cc:1059] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:58.014950 31234 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:58.016393 31185 raft_consensus.cc:3055] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:03:58.016933 31185 raft_consensus.cc:740] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:58.019887 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.020983 30603 raft_consensus.cc:2393] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:58.020534 31234 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.022524 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.023526 30599 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:58.011006 30675 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.024219 31234 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.019291 31185 consensus_queue.cc:260] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.025686 31185 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.026085 30596 raft_consensus.cc:1275] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:58.025993 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.026944 30600 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:58.027256 31380 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:58.028088 31380 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.032444 31380 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.033215 31268 consensus_queue.cc:1059] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:58.001636 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:58.034339 31262 raft_consensus.cc:3055] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:03:58.034904 31262 raft_consensus.cc:740] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:58.035197 30674 raft_consensus.cc:2393] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:58.036489 31262 consensus_queue.cc:260] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:57.999042 31278 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:58.029641 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 48133.6us
I20251025 20:03:57.993851 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:58.046100 30671 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.055867 30796 tablet_bootstrap.cc:492] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:58.056806 30796 ts_tablet_manager.cc:1403] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.644s user 0.006s sys 0.029s
I20251025 20:03:58.056404 31385 consensus_queue.cc:1048] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:58.060073 31259 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.058506 31262 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.060843 31259 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.061354 30598 raft_consensus.cc:1275] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:58.062508 31242 raft_consensus.cc:3055] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:03:58.062954 31242 raft_consensus.cc:740] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:58.063346 30597 raft_consensus.cc:1275] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:58.064100 31242 consensus_queue.cc:260] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.065711 31242 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.066309 31259 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:58.067174 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 46440.5us
I20251025 20:03:58.079605 30570 leader_election.cc:400] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3.
I20251025 20:03:58.080011 30570 leader_election.cc:403] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:03:58.079433 30500 catalog_manager.cc:5649] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 0 to 1, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 1 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:03:58.080961 30570 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.067536 30796 raft_consensus.cc:359] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.085335 30796 raft_consensus.cc:385] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:58.086604 30796 raft_consensus.cc:740] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:58.081790 31259 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.087862 30796 consensus_queue.cc:260] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.089996 31278 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:58.087564 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:03:58.097826 30716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:58.088201 31185 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.104008 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.095961 31278 raft_consensus.cc:697] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:58.095072 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.100952 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:58.102385 30746 raft_consensus.cc:3060] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:58.111099 30793 tablet_bootstrap.cc:492] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:58.109819 30501 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39186 (request call id 97) took 4601 ms. Trace:
I20251025 20:03:58.106832 30600 raft_consensus.cc:2410] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:03:58.104003 30673 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:58.118018 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.096150 31259 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.116398 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.107313 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.117223 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.129189 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:58.124562 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.130460 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.115559 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.118772 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.132292 30670 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:58.139565 30744 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:58.140133 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.140404 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.127571 30675 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:58.108475 31278 consensus_queue.cc:237] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.142799 30752 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:03:58.143188 30796 ts_tablet_manager.cc:1434] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.086s user 0.001s sys 0.010s
I20251025 20:03:58.129483 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:58.138404 30672 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:58.149019 30678 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:58.148409 30595 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:03:58.154040 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.146219 30599 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.150872 30745 raft_consensus.cc:2393] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:03:58.154276 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.154996 30793 ts_tablet_manager.cc:1403] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.485s user 0.024s sys 0.039s
I20251025 20:03:58.152769 30749 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:58.165010 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.123086 30501 rpcz_store.cc:276] 1025 20:03:53.508614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:53.511464 (+ 2850us) service_pool.cc:225] Handling call
1025 20:03:58.109562 (+4598098us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":319369,"reported_tablets":12}
I20251025 20:03:58.154951 30747 raft_consensus.cc:3060] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:58.171386 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:58.184145 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.185446 30796 tablet_bootstrap.cc:492] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:03:58.182408 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.190577 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:58.192286 30752 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:58.200100 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:58.201238 30719 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.201524 30678 raft_consensus.cc:2393] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:58.194523 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.207679 30674 raft_consensus.cc:2393] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:58.214524 30603 raft_consensus.cc:1275] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:58.215617 31259 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:58.216408 30677 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:58.217638 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.204552 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.226493 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.234784 31259 raft_consensus.cc:2804] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:58.235304 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.237468 30671 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:58.240206 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.242362 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.250034 30595 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:58.224448 30673 raft_consensus.cc:1275] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:58.264150 30800 tablet_bootstrap.cc:492] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:58.264117 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.266142 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:58.251614 31259 raft_consensus.cc:697] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:58.271306 31185 raft_consensus.cc:2749] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3.
I20251025 20:03:58.273083 30793 raft_consensus.cc:359] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.274164 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.278735 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.281030 30793 raft_consensus.cc:385] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:58.282567 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.281358 30793 raft_consensus.cc:740] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:03:58.283115 30673 raft_consensus.cc:1275] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:58.283715 30748 raft_consensus.cc:2468] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:58.283643 30793 consensus_queue.cc:260] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.285260 31259 consensus_queue.cc:237] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.285823 30793 ts_tablet_manager.cc:1434] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.127s user 0.006s sys 0.001s
I20251025 20:03:58.286312 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.299603 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.299397 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.301172 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.301730 30719 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.300464 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.303587 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.303975 31209 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:58.304435 31209 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:58.304886 31209 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:58.305348 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.306140 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.306859 31385 consensus_queue.cc:1048] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:58.307036 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:58.308060 30601 raft_consensus.cc:2368] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3.
I20251025 20:03:58.306989 30750 raft_consensus.cc:2393] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:58.308388 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.310053 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.310839 31384 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.311411 31384 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.313200 31290 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.313845 31290 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.314760 31192 consensus_queue.cc:1048] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:58.316191 31290 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.316817 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.318991 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.313736 31384 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.319722 30599 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:58.320600 30602 raft_consensus.cc:1240] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:58.321600 30800 ts_tablet_manager.cc:1403] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.538s user 0.035s sys 0.039s
I20251025 20:03:58.327733 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.329417 30567 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.328830 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.330610 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:58.333428 30678 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.329752 30673 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:58.340358 30793 tablet_bootstrap.cc:492] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
W20251025 20:03:58.342526 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:58.343269 30567 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.344214 31242 raft_consensus.cc:2749] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:03:58.344746 30567 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.346004 31185 raft_consensus.cc:2749] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.346953 31242 raft_consensus.cc:2749] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:03:58.350785 31185 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:58.351411 31185 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.353472 31185 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:58.358291 30567 leader_election.cc:343] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:58.358122 30716 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:58.360020 31385 raft_consensus.cc:2749] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:03:58.367667 30716 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:58.368191 30596 raft_consensus.cc:2468] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:58.368767 31385 raft_consensus.cc:2749] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:03:58.346393 31278 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.372583 31278 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.374135 30719 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.375258 31385 raft_consensus.cc:2804] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:58.375705 31385 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:58.376119 31385 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.376250 31262 consensus_queue.cc:1048] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:58.376621 31325 consensus_queue.cc:1048] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:58.378394 30719 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.369506 30716 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:58.386868 31262 raft_consensus.cc:2804] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:58.391266 31262 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:58.391673 31262 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:58.394502 30645 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.395367 31384 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:03:58.395748 31384 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:58.396136 31384 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:03:58.365139 30679 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
W20251025 20:03:58.403759 30719 leader_election.cc:343] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:58.404417 30719 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.406081 30675 raft_consensus.cc:1275] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:58.411800 30746 raft_consensus.cc:1275] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:03:58.414036 31268 raft_consensus.cc:2749] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:03:58.415287 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 45929.3us
I20251025 20:03:58.419700 30716 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:58.420639 31325 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:58.421023 30747 raft_consensus.cc:2468] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:03:58.421144 31325 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:03:58.405190 30570 leader_election.cc:343] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:58.426985 31325 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.427027 31268 raft_consensus.cc:2804] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:58.427839 31268 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:58.428198 31268 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.406148 31385 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.428637 30570 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.429558 31242 raft_consensus.cc:2804] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:58.430009 31242 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:58.430290 31385 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:58.430387 31242 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.433987 31290 raft_consensus.cc:493] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.433643 30800 raft_consensus.cc:359] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.434516 30800 raft_consensus.cc:385] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:58.434636 31185 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.434535 31290 raft_consensus.cc:515] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.434851 30800 raft_consensus.cc:740] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:58.435396 31185 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.436949 31290 leader_election.cc:290] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.436039 30800 consensus_queue.cc:260] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.438535 30796 tablet_bootstrap.cc:654] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:58.437568 31185 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.445194 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.440315 30719 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.441102 30570 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.442984 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.443832 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.444237 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.444809 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.445055 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:58.447273 30645 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.451538 30676 raft_consensus.cc:2368] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3.
I20251025 20:03:58.452308 30570 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.452582 31281 raft_consensus.cc:2804] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:58.452929 31281 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:58.453277 31281 raft_consensus.cc:3060] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:58.454137 30567 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.454779 30793 tablet_bootstrap.cc:654] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:58.456811 30716 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:58.457882 31234 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:58.458340 31234 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:58.458753 31234 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.458477 31384 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.462095 31391 consensus_queue.cc:1048] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:58.464424 31278 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.447821 30800 ts_tablet_manager.cc:1434] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.126s user 0.008s sys 0.000s
I20251025 20:03:58.476878 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.477797 30678 raft_consensus.cc:2468] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:58.482679 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.473409 31185 raft_consensus.cc:2804] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:58.485512 30570 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:03:58.496238 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:58.480433 31278 raft_consensus.cc:2764] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20251025 20:03:58.503527 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 49515.1us
I20251025 20:03:58.502967 31384 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.502785 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:03:58.506783 30570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:58.507686 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.509102 30567 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.491317 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.492446 31185 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:03:58.514905 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 91069.8us
I20251025 20:03:58.491787 31259 raft_consensus.cc:2749] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:03:58.515974 30719 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.490455 31209 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.522208 30800 tablet_bootstrap.cc:492] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:58.514575 31185 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:58.499855 31242 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.518106 31278 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:58.526301 31181 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:58.531808 30746 raft_consensus.cc:2468] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:03:58.531975 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.542749 30796 tablet_bootstrap.cc:492] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:58.543277 30796 ts_tablet_manager.cc:1403] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.381s user 0.017s sys 0.009s
I20251025 20:03:58.544498 31209 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:58.546200 30567 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.547039 31413 raft_consensus.cc:2804] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:58.547626 31413 raft_consensus.cc:697] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:03:58.547681 30567 leader_election.cc:343] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:58.548192 30567 leader_election.cc:304] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:58.549289 30567 leader_election.cc:343] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:58.548683 31413 consensus_queue.cc:237] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.550096 31378 raft_consensus.cc:2804] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:58.550565 31378 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:58.550916 31378 raft_consensus.cc:3060] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:58.556716 31181 raft_consensus.cc:697] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:58.558250 31181 consensus_queue.cc:237] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.559286 31242 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.559965 31242 raft_consensus.cc:2749] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:58.564163 30716 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:58.523583 31299 raft_consensus.cc:2804] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:58.565725 31299 raft_consensus.cc:697] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:58.566550 30567 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.566864 31299 consensus_queue.cc:237] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.578352 31268 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.579399 31414 raft_consensus.cc:2804] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:58.580155 31414 raft_consensus.cc:697] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:58.580765 31262 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.533445 31278 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:58.594149 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.594946 30596 raft_consensus.cc:1275] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:58.595199 30601 raft_consensus.cc:2393] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:58.589253 31278 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.608649 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:58.609071 31262 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:58.610352 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 80248.5us
I20251025 20:03:58.599679 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:58.626933 30570 leader_election.cc:343] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:58.591516 31325 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.630867 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:58.631500 30745 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.635056 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.636132 31420 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:58.636591 31420 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:58.636971 31420 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:58.637362 31268 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:58.630884 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:58.638417 31234 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.640923 31234 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:58.642024 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:58.582345 31281 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.651175 30745 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
W20251025 20:03:58.658787 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 142853us
I20251025 20:03:58.659847 30796 raft_consensus.cc:359] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.660851 30796 raft_consensus.cc:385] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:58.661193 30796 raft_consensus.cc:740] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:58.666370 30716 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:58.663746 30645 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.667871 30796 consensus_queue.cc:260] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.630097 30601 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
I20251025 20:03:58.668570 30570 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.658819 31290 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.581400 31414 consensus_queue.cc:237] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.703037 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:03:58.703750 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 44724.8us
I20251025 20:03:58.706082 30603 raft_consensus.cc:2368] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:58.707557 30645 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.691123 31327 raft_consensus.cc:2804] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:58.708550 30796 ts_tablet_manager.cc:1434] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.165s user 0.008s sys 0.002s
I20251025 20:03:58.735332 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:58.748144 30603 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.748863 31192 raft_consensus.cc:2749] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.759960 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:58.762102 30750 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.706488 31385 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:03:58.748147 31185 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.710656 31384 raft_consensus.cc:2749] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:03:58.727588 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:58.697825 31290 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.726631 31281 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.770506 30796 tablet_bootstrap.cc:492] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
W20251025 20:03:58.772303 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 64837.1us
I20251025 20:03:58.754904 31327 raft_consensus.cc:697] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:58.726620 31325 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:03:58.772634 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 83846us
I20251025 20:03:58.791438 30793 tablet_bootstrap.cc:492] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:58.793264 30793 ts_tablet_manager.cc:1403] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.461s user 0.025s sys 0.039s
I20251025 20:03:58.767163 31385 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:58.796974 30645 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.731125 31420 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.732352 31378 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.790489 30601 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.725570 31309 consensus_queue.cc:1048] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20251025 20:03:58.808946 30716 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:58.810719 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:58.814420 30645 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.816112 31318 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.817214 30800 tablet_bootstrap.cc:654] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:58.816838 31327 consensus_queue.cc:237] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.800355 31385 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:03:58.820065 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 253595us
I20251025 20:03:58.816316 30645 leader_election.cc:304] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.809790 31185 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.811720 31384 raft_consensus.cc:2749] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.840260 30645 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.847064 30750 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:03:58.847862 30567 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.818320 31415 raft_consensus.cc:493] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.851465 30745 raft_consensus.cc:1275] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:03:58.852340 30570 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.854218 31192 raft_consensus.cc:2749] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.861877 30603 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:58.862812 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:58.865373 31415 raft_consensus.cc:515] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.885423 31384 raft_consensus.cc:2749] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.863576 30793 raft_consensus.cc:359] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.890069 31429 raft_consensus.cc:2749] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:03:58.896121 30793 raft_consensus.cc:385] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 20:03:58.896224 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 43974.9us
I20251025 20:03:58.898787 30793 raft_consensus.cc:740] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
W20251025 20:03:58.901367 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 53643.2us
I20251025 20:03:58.861370 31378 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.899235 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:58.904863 31278 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:58.906080 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 65933.5us
W20251025 20:03:58.907182 30645 leader_election.cc:343] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:58.904654 30793 consensus_queue.cc:260] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:58.899139 31419 log.cc:927] Time spent T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20251025 20:03:58.915263 30793 ts_tablet_manager.cc:1434] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.122s user 0.009s sys 0.006s
I20251025 20:03:58.916282 31427 consensus_queue.cc:1048] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20251025 20:03:58.917095 30716 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:58.917188 31234 consensus_queue.cc:1048] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:03:58.917479 30567 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.918454 31433 raft_consensus.cc:2804] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:58.919005 31433 raft_consensus.cc:697] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:58.910991 31435 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:58.920004 31433 consensus_queue.cc:237] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.920725 31435 raft_consensus.cc:697] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:58.921813 31435 consensus_queue.cc:237] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.922531 31416 raft_consensus.cc:2749] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:03:58.922214 30567 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.923950 31432 raft_consensus.cc:2749] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.926055 31389 raft_consensus.cc:2749] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.933200 31278 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.934610 30599 raft_consensus.cc:1275] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:58.937901 31318 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.944046 30645 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.946918 30793 tablet_bootstrap.cc:492] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:03:58.911316 31185 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:58.938509 31318 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.950668 31309 raft_consensus.cc:2749] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.951889 31318 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.955422 30567 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.954157 30645 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.960055 30645 leader_election.cc:400] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3.
I20251025 20:03:58.960682 30645 leader_election.cc:403] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251025 20:03:58.966353 30719 leader_election.cc:400] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:58.966192 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:58.966683 30719 leader_election.cc:403] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251025 20:03:58.968861 30567 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.972267 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:58.973126 30750 raft_consensus.cc:2368] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3.
I20251025 20:03:58.959571 31384 raft_consensus.cc:2749] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:03:58.979291 30716 leader_election.cc:343] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:58.979507 30570 leader_election.cc:400] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3.
I20251025 20:03:58.979722 30716 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:58.980052 30570 leader_election.cc:403] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251025 20:03:58.967252 31278 raft_consensus.cc:2749] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.978598 30719 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:58.972570 31281 raft_consensus.cc:2749] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3.
I20251025 20:03:58.984771 30645 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.957119 31185 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.973701 31414 raft_consensus.cc:2749] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:58.973119 31420 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:58.986838 30645 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:58.977726 31389 raft_consensus.cc:2749] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:58.986925 30567 leader_election.cc:400] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975) with higher term. Message: Invalid argument: T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3.
I20251025 20:03:58.990231 30567 leader_election.cc:403] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:03:58.993014 31416 raft_consensus.cc:2804] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:58.993561 31416 raft_consensus.cc:697] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:58.987491 31327 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:58.987274 31318 raft_consensus.cc:2749] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:03:59.000550 31416 consensus_queue.cc:237] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:58.991711 31414 raft_consensus.cc:2749] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:59.009435 30645 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.001595 31281 raft_consensus.cc:2749] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:03:59.002833 31278 raft_consensus.cc:2749] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975) with higher term. Message: Invalid argument: T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3.
I20251025 20:03:59.016415 31309 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:59.027079 31318 consensus_queue.cc:1059] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:03:59.028362 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:59.028641 31318 raft_consensus.cc:3055] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:59.029093 30749 raft_consensus.cc:2468] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:59.029206 31318 raft_consensus.cc:740] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:59.003854 31415 leader_election.cc:290] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:59.017525 30567 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.019572 31440 consensus_queue.cc:1048] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20251025 20:03:59.020295 31428 consensus_queue.cc:1048] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20251025 20:03:59.035717 30570 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:59.039760 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.046211 30678 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:59.042433 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.046888 30676 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:59.048262 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:59.048950 30672 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:03:59.049830 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.050083 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.050715 30672 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:59.051306 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.056598 30678 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
W20251025 20:03:59.047600 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 46810.7us
I20251025 20:03:59.059801 30567 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.045538 31278 raft_consensus.cc:2749] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251025 20:03:59.062172 30645 leader_election.cc:343] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:59.027086 31185 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.062440 30567 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.060906 31192 consensus_queue.cc:1048] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251025 20:03:59.063284 31234 raft_consensus.cc:2804] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:59.080544 31234 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:59.081055 31234 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:03:59.067947 30645 leader_election.cc:343] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:59.071003 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:59.086593 30677 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:03:59.055420 31420 raft_consensus.cc:2804] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:59.087837 31420 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:59.088335 31420 raft_consensus.cc:3060] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:59.089325 30500 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39172 (request call id 89) took 3400 ms. Trace:
I20251025 20:03:59.061743 31268 consensus_queue.cc:1048] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20251025 20:03:59.074699 31318 consensus_queue.cc:260] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.075830 30601 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:59.076550 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:59.093839 30678 raft_consensus.cc:2393] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:59.078879 30796 tablet_bootstrap.cc:654] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:59.098518 31185 raft_consensus.cc:2749] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251025 20:03:59.104612 31442 log.cc:927] Time spent T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20251025 20:03:59.106990 30674 raft_consensus.cc:1240] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: []
I20251025 20:03:59.111003 30645 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:03:59.112152 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:03:59.122459 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:59.135826 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:59.124142 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:59.098547 31318 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:59.089821 30673 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:59.138932 31290 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.112140 31192 raft_consensus.cc:2804] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:59.139724 31385 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.141657 30745 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:59.140779 31192 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:59.153051 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:59.153738 30750 raft_consensus.cc:3060] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:59.154268 30676 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:03:59.155638 30745 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
W20251025 20:03:59.157543 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 90849.7us
I20251025 20:03:59.156864 31192 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:59.163915 30800 tablet_bootstrap.cc:492] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:03:59.164480 30800 ts_tablet_manager.cc:1403] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.650s user 0.013s sys 0.011s
W20251025 20:03:59.166095 30567 consensus_peers.cc:597] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 -> Peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Couldn't send request to peer 583af532dff04e8ebba45013323d516d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:03:59.166844 30567 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.167213 30793 tablet_bootstrap.cc:654] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
W20251025 20:03:59.170457 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 191181us
I20251025 20:03:59.175383 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:59.178745 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:59.117468 31234 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.146746 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.187012 30750 raft_consensus.cc:2468] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:03:59.143618 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:59.135710 30672 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:03:59.142591 31385 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:59.204690 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:59.206678 30753 raft_consensus.cc:2468] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 0.
W20251025 20:03:59.209784 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 51496.4us
I20251025 20:03:59.211696 30501 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39160 (request call id 93) took 1013 ms. Trace:
I20251025 20:03:59.212118 30501 rpcz_store.cc:276] 1025 20:03:58.197288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:58.246244 (+ 48956us) service_pool.cc:225] Handling call
1025 20:03:59.206170 (+959926us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":15}
I20251025 20:03:59.212641 30567 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:59.213590 30716 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:59.198294 31309 raft_consensus.cc:2749] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:03:59.202322 31185 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:03:59.220432 31185 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:59.182798 31420 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.205868 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:59.222268 30567 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.221052 31185 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:59.089766 30500 rpcz_store.cc:276] 1025 20:03:55.694026 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:55.694205 (+ 179us) service_pool.cc:225] Handling call
1025 20:03:59.089239 (+3395034us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 20:03:58.252855 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 20:03:58.287012 (+ 34157us) write_op.cc:432] Acquiring schema lock in shared mode
1025 20:03:58.287071 (+ 59us) write_op.cc:435] Acquired schema lock
1025 20:03:58.287095 (+ 24us) tablet.cc:660] Decoding operations
1025 20:03:58.287585 (+ 490us) write_op.cc:620] Acquiring the partition lock for write op
1025 20:03:58.287696 (+ 111us) write_op.cc:641] Partition lock acquired for write op
1025 20:03:58.287743 (+ 47us) tablet.cc:683] Acquiring locks for 1 operations
1025 20:03:58.288074 (+ 331us) tablet.cc:699] Row locks acquired
1025 20:03:58.288110 (+ 36us) write_op.cc:260] PREPARE: finished
1025 20:03:58.288307 (+ 197us) write_op.cc:270] Start()
1025 20:03:58.288496 (+ 189us) write_op.cc:276] Timestamp: P: 1761422638288245 usec, L: 0
1025 20:03:58.288538 (+ 42us) op_driver.cc:348] REPLICATION: starting
1025 20:03:58.289274 (+ 736us) log.cc:844] Serialized 543 byte log entry
1025 20:03:58.609654 (+320380us) op_driver.cc:464] REPLICATION: finished
1025 20:03:58.754179 (+144525us) write_op.cc:301] APPLY: starting
1025 20:03:58.754274 (+ 95us) tablet.cc:1365] starting BulkCheckPresence
1025 20:03:58.754403 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
1025 20:03:58.768898 (+ 14495us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 20:03:58.769302 (+ 404us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 20:03:58.769581 (+ 279us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 20:03:58.769651 (+ 70us) write_op.cc:312] APPLY: finished
1025 20:03:58.830744 (+ 61093us) log.cc:844] Serialized 22 byte log entry
1025 20:03:58.853112 (+ 22368us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:03:58.853223 (+ 111us) write_op.cc:454] Released schema lock
1025 20:03:58.853286 (+ 63us) write_op.cc:341] FINISH: Updating metrics
1025 20:03:58.853750 (+ 464us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:03:58.853784 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1426398,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":144243,"apply.run_cpu_time_us":3369,"apply.run_wall_time_us":99497,"num_ops":1,"prepare.queue_time_us":70857,"prepare.run_cpu_time_us":10241,"prepare.run_wall_time_us":221096,"raft.queue_time_us":132581,"raft.run_cpu_time_us":24834,"raft.run_wall_time_us":154862,"replication_time_us":320949,"thread_start_us":439959,"threads_started":4,"wal-append.queue_time_us":175658}]]}
I20251025 20:03:59.215108 31299 consensus_queue.cc:1059] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.013s
I20251025 20:03:59.227708 30679 raft_consensus.cc:1275] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:03:59.228152 31299 raft_consensus.cc:3055] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:03:59.228622 31299 raft_consensus.cc:740] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:59.229028 30751 raft_consensus.cc:1275] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.234630 31435 consensus_queue.cc:1048] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251025 20:03:59.236996 31420 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:59.242261 30645 leader_election.cc:343] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:59.243639 31234 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.242913 30645 leader_election.cc:304] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.244302 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:59.245085 31380 raft_consensus.cc:2749] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 20:03:59.247028 30645 leader_election.cc:343] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:59.247713 30645 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.248324 30595 raft_consensus.cc:2393] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:59.248867 31380 raft_consensus.cc:2749] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:03:59.250296 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.251120 30675 raft_consensus.cc:2393] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:59.230443 31389 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:59.251996 31389 raft_consensus.cc:697] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:59.252944 31389 consensus_queue.cc:237] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:59.258560 30645 leader_election.cc:343] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:59.259868 30645 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:03:59.263036 30719 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:03:59.266335 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 52816.6us
I20251025 20:03:59.269188 31192 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.270085 31268 raft_consensus.cc:2749] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:03:59.274227 30645 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:59.271622 30800 raft_consensus.cc:359] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.276028 30800 raft_consensus.cc:385] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:59.275943 30645 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:59.276409 30800 raft_consensus.cc:740] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:03:59.270546 31428 consensus_queue.cc:1048] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.007s
I20251025 20:03:59.229508 31299 consensus_queue.cc:260] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.280496 31299 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:03:59.281690 31185 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.270745 31380 raft_consensus.cc:2804] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:03:59.287925 30800 consensus_queue.cc:260] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.287037 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:59.293139 31431 raft_consensus.cc:2749] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:59.284159 31391 raft_consensus.cc:2804] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:03:59.302436 30748 raft_consensus.cc:2393] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:59.303038 31418 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:59.288861 31259 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:03:59.321187 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 58232.9us
I20251025 20:03:59.324282 30800 ts_tablet_manager.cc:1434] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.159s user 0.009s sys 0.000s
I20251025 20:03:59.305581 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.290556 31380 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:59.331590 30570 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:03:59.332583 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 55565.9us
I20251025 20:03:59.326470 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:59.340499 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.341136 30676 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:59.330473 31455 raft_consensus.cc:2804] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:59.323514 31391 raft_consensus.cc:697] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:59.335338 30672 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:59.335455 31431 raft_consensus.cc:2749] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:03:59.322459 31290 consensus_queue.cc:1048] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251025 20:03:59.335669 31185 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.335088 31380 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:03:59.365401 30570 leader_election.cc:343] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:59.327381 31259 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.322688 31418 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.399592 30676 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:59.379590 30800 tablet_bootstrap.cc:492] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:03:59.404320 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:59.351995 31455 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:59.414328 30603 raft_consensus.cc:1240] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:03:59.389519 31448 raft_consensus.cc:2804] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:59.392477 30604 raft_consensus.cc:1275] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.387876 31192 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.369606 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:03:59.385432 31460 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:03:59.442827 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.371678 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:59.452044 30676 raft_consensus.cc:2368] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:59.429019 31455 raft_consensus.cc:3060] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:59.432763 31185 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:59.463814 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.464744 30678 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:59.478490 30603 raft_consensus.cc:1240] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:59.452493 31448 raft_consensus.cc:697] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:59.469462 31185 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.472913 31181 consensus_queue.cc:1059] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.020s
W20251025 20:03:59.493198 31458 log.cc:927] Time spent T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.145s user 0.002s sys 0.000s
I20251025 20:03:59.485813 31418 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:59.492376 30751 raft_consensus.cc:1275] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.444820 31391 consensus_queue.cc:237] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.520841 31423 consensus_queue.cc:1048] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20251025 20:03:59.453631 31460 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.541940 31460 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:59.543287 31456 log.cc:927] Time spent T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20251025 20:03:59.560225 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:03:59.560467 31327 consensus_meta.cc:287] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.557s user 0.011s sys 0.001s
I20251025 20:03:59.561020 30598 raft_consensus.cc:2468] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:59.561406 30602 raft_consensus.cc:1275] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.564057 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:59.561028 31327 raft_consensus.cc:2749] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3.
I20251025 20:03:59.566220 31448 consensus_queue.cc:237] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.574343 30676 raft_consensus.cc:1275] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.578161 31185 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.586469 31185 consensus_queue.cc:1048] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:59.586784 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:59.590082 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:59.590276 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:59.590793 30745 raft_consensus.cc:2468] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
W20251025 20:03:59.593915 30716 leader_election.cc:343] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:59.597522 30597 raft_consensus.cc:1275] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.600833 30716 leader_election.cc:400] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Vote denied by peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975) with higher term. Message: Invalid argument: T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:59.601287 30716 leader_election.cc:403] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251025 20:03:59.602391 30719 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:59.605588 30678 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:59.609077 30570 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:03:59.610389 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 214090us
I20251025 20:03:59.611168 30645 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.587183 30596 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:59.613406 30745 raft_consensus.cc:1275] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.577801 31259 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:59.579171 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:03:59.633283 30719 leader_election.cc:343] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:03:59.635365 30719 leader_election.cc:343] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:59.639778 30719 leader_election.cc:304] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:59.631340 31185 raft_consensus.cc:2804] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:03:59.645620 30679 raft_consensus.cc:1275] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.639683 31299 consensus_queue.cc:1048] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20251025 20:03:59.646858 30716 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:03:59.634505 31423 raft_consensus.cc:2804] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251025 20:03:59.667135 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 56094.3us
I20251025 20:03:59.645788 31185 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:59.608284 31389 consensus_queue.cc:1048] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20251025 20:03:59.643582 31378 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:59.642105 31262 raft_consensus.cc:2749] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975) with higher term. Message: Invalid argument: T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:59.639345 31428 raft_consensus.cc:3055] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:03:59.667568 30673 raft_consensus.cc:2368] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:59.661952 31414 raft_consensus.cc:2749] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 20:03:59.687927 30645 leader_election.cc:343] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:03:59.644548 31418 consensus_queue.cc:1048] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20251025 20:03:59.661602 31309 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:03:59.684041 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:03:59.664112 31423 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:59.659474 31427 consensus_queue.cc:1048] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20251025 20:03:59.676801 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:59.729409 30716 leader_election.cc:400] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Vote denied by peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975) with higher term. Message: Invalid argument: T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3.
I20251025 20:03:59.689265 31185 raft_consensus.cc:3060] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:03:59.702354 31428 raft_consensus.cc:740] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:59.702512 30672 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:03:59.683462 31461 raft_consensus.cc:2749] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251025 20:03:59.701848 31378 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.676700 31415 consensus_queue.cc:1048] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20251025 20:03:59.737299 30567 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.738219 30676 raft_consensus.cc:1240] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:03:59.729758 31435 consensus_queue.cc:1048] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20251025 20:03:59.743412 31378 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.730355 31423 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:59.749102 30796 tablet_bootstrap.cc:492] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:03:59.749783 30796 ts_tablet_manager.cc:1403] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.985s user 0.016s sys 0.037s
W20251025 20:03:59.744535 31463 log.cc:927] Time spent T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20251025 20:03:59.736250 30679 raft_consensus.cc:2468] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:03:59.758015 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:59.758275 31380 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.758821 30679 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:03:59.760181 30567 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:03:59.760416 31380 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.732810 31309 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.761330 30679 raft_consensus.cc:1275] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.761570 31431 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:03:59.762027 31431 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:03:59.762351 31431 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:03:59.762840 31309 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.765102 31445 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:59.765532 30600 raft_consensus.cc:1240] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:59.766155 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.766724 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.769982 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:03:59.770659 30674 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:03:59.766978 30675 raft_consensus.cc:2393] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:03:59.730697 31460 consensus_queue.cc:1059] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20251025 20:03:59.775777 31462 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:59.776340 31462 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.777549 30675 raft_consensus.cc:1275] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.780283 30716 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:03:59.777807 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:59.778371 31446 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:03:59.773058 31389 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:59.794252 31389 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.795198 31460 raft_consensus.cc:3055] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:03:59.795619 31460 raft_consensus.cc:740] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:03:59.796291 31461 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:03:59.796434 31262 raft_consensus.cc:2749] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:03:59.796463 31389 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.796519 31460 consensus_queue.cc:260] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.796829 31461 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.803333 30674 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:03:59.773128 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:59.744781 31420 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:03:59.782047 30567 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.778694 31462 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.744994 31259 consensus_queue.cc:1048] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
W20251025 20:03:59.783535 31471 log.cc:927] Time spent T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20251025 20:03:59.829656 30750 raft_consensus.cc:1240] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:03:59.834344 31413 consensus_queue.cc:1048] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:03:59.765618 31445 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.811092 31385 consensus_queue.cc:1048] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20251025 20:03:59.841398 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:59.842204 31445 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:03:59.822276 31389 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:03:59.844419 31389 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.821110 30748 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:03:59.822556 31420 raft_consensus.cc:697] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:03:59.853471 31420 consensus_queue.cc:237] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.854420 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:03:59.799171 31461 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:03:59.797845 31460 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:03:59.794195 30670 raft_consensus.cc:1275] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.846498 31435 raft_consensus.cc:2749] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:03:59.809182 31446 raft_consensus.cc:697] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:03:59.844746 30796 raft_consensus.cc:359] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.865774 30796 raft_consensus.cc:385] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:03:59.866223 30796 raft_consensus.cc:740] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:03:59.821556 31428 consensus_queue.cc:260] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.777823 31185 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.822052 30673 raft_consensus.cc:2468] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
W20251025 20:03:59.873775 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 209084us
I20251025 20:03:59.870582 30796 consensus_queue.cc:260] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.872460 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:03:59.879318 30670 raft_consensus.cc:1275] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.880475 30603 raft_consensus.cc:1275] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251025 20:03:59.882712 30603 pending_rounds.cc:85] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957: Aborting all ops after (but not including) 1
I20251025 20:03:59.883283 30603 pending_rounds.cc:107] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251025 20:03:59.883594 30603 raft_consensus.cc:2889] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:03:59.883917 30673 raft_consensus.cc:1275] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.884269 30800 tablet_bootstrap.cc:654] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:03:59.856428 30675 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:03:59.888652 31446 consensus_queue.cc:237] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.891762 31378 consensus_queue.cc:1048] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:03:59.894142 31454 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:03:59.895234 30793 tablet_bootstrap.cc:492] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:03:59.895613 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:59.903285 30793 ts_tablet_manager.cc:1403] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.977s user 0.044s sys 0.081s
I20251025 20:03:59.896840 31389 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:03:59.900076 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 191582us
I20251025 20:03:59.887602 31435 consensus_queue.cc:1048] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20251025 20:03:59.885285 30673 raft_consensus.cc:1275] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:03:59.926964 30716 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
W20251025 20:03:59.928941 30645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:03:59.923975 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:03:59.880332 31455 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:03:59.931701 30645 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:03:59.934530 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:03:59.903525 31299 consensus_queue.cc:1048] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20251025 20:03:59.947708 30796 ts_tablet_manager.cc:1434] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.197s user 0.009s sys 0.000s
I20251025 20:03:59.950387 30603 raft_consensus.cc:2393] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:03:59.962555 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:59.962771 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:03:59.964660 31431 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:03:59.987895 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 56316.1us
W20251025 20:03:59.994212 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 68194.2us
I20251025 20:04:00.006192 30671 raft_consensus.cc:2393] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
W20251025 20:04:00.011308 31480 log.cc:927] Time spent T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20251025 20:03:59.939603 31454 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.014005 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:00.014894 30599 raft_consensus.cc:2368] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:00.020094 31385 raft_consensus.cc:2804] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:04:00.015255 31481 raft_consensus.cc:2749] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:00.079097 30796 tablet_bootstrap.cc:492] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
W20251025 20:04:00.079808 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:00.027786 31290 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:00.027691 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:00.033057 31455 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:00.021525 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:00.099298 31486 log.cc:927] Time spent T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20251025 20:04:00.076336 31385 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:00.108135 30793 raft_consensus.cc:359] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.132462 30793 raft_consensus.cc:385] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:00.063832 31262 consensus_queue.cc:1059] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20251025 20:04:00.132951 30793 raft_consensus.cc:740] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
W20251025 20:04:00.135747 30645 leader_election.cc:343] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:04:00.139216 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:00.139901 30748 raft_consensus.cc:3060] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:04:00.140066 31441 log.cc:927] Time spent T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20251025 20:04:00.140799 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:00.142181 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:00.105329 31185 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:00.096012 31290 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.138309 30793 consensus_queue.cc:260] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.161082 30645 leader_election.cc:400] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:00.235374 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:00.296296 30645 leader_election.cc:403] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:04:00.297346 31470 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 4
I20251025 20:04:00.145231 31431 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:00.308904 31454 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:00.103374 31445 consensus_queue.cc:1048] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.139s
W20251025 20:04:00.236907 31479 log.cc:927] Time spent T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.143s user 0.000s sys 0.000s
I20251025 20:04:00.111714 31385 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:04:00.334339 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 187714us
I20251025 20:04:00.334316 30599 raft_consensus.cc:2393] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:00.308971 31413 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:00.360092 31485 raft_consensus.cc:493] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:00.367224 30645 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:00.383332 30793 ts_tablet_manager.cc:1434] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.480s user 0.015s sys 0.020s
I20251025 20:04:00.384857 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:00.385771 30745 raft_consensus.cc:2468] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:00.380844 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:00.388401 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 461499us
I20251025 20:04:00.388530 30645 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:00.387428 30602 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:00.094970 31423 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.401350 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:00.384994 31454 raft_consensus.cc:2749] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:00.402319 30745 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:00.408491 30645 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:04:00.410705 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 388105us
W20251025 20:04:00.508679 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:00.362557 31485 raft_consensus.cc:515] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.377842 31413 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.547202 30678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 414) took 1195 ms. Trace:
I20251025 20:04:00.551498 30678 rpcz_store.cc:276] 1025 20:03:59.352353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.607936 (+255583us) service_pool.cc:225] Handling call
1025 20:03:59.642551 (+ 34615us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:59.643005 (+ 454us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:59.643049 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:03:59.662585 (+ 19536us) log.cc:844] Serialized 26 byte log entry
1025 20:03:59.856757 (+194172us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:03:59.856874 (+ 117us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:59.857776 (+ 902us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:00.546861 (+689085us) raft_consensus.cc:1685] finished
1025 20:04:00.546895 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:00.547115 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":18722,"raft.run_cpu_time_us":384,"raft.run_wall_time_us":17914,"thread_start_us":193628,"threads_started":1,"wal-append.queue_time_us":328864}
I20251025 20:04:00.556444 31470 raft_consensus.cc:2749] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:00.419258 31460 raft_consensus.cc:2804] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:00.419466 31281 raft_consensus.cc:2804] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:00.420611 31455 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:00.630352 31281 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:00.630535 31460 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:00.557329 31478 consensus_queue.cc:1048] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.236s
I20251025 20:04:00.659732 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:00.668583 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:00.668869 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 256361us
W20251025 20:04:00.681524 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 263906us
W20251025 20:03:59.874917 31418 consensus_queue.cc:1175] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "583af532dff04e8ebba45013323d516d" responder_term: 4 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20251025 20:04:00.660970 30672 raft_consensus.cc:1275] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:00.662798 31281 raft_consensus.cc:3060] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:00.634939 31455 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.666687 31460 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:00.710426 31493 consensus_queue.cc:1048] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20251025 20:04:00.711550 31413 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:00.717489 31489 log.cc:927] Time spent T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251025 20:04:00.719419 30800 tablet_bootstrap.cc:492] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:04:00.719952 30800 ts_tablet_manager.cc:1403] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 1.371s user 0.009s sys 0.027s
I20251025 20:04:00.697666 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:00.736680 30601 raft_consensus.cc:2393] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:00.732842 31290 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:00.740201 30796 tablet_bootstrap.cc:654] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
W20251025 20:04:00.662130 30751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.037s sys 0.042s
W20251025 20:04:00.750454 30751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.785s user 0.037s sys 0.042s
I20251025 20:04:00.782636 31378 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:00.783491 31378 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.784543 31488 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:00.785104 31488 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.786103 31378 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:00.786347 31485 leader_election.cc:290] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:00.787364 31488 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:00.788348 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:00.790416 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:00.798105 31455 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:00.746217 31423 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:00.801003 31327 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:00.802726 31492 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:00.803141 31327 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.803336 31492 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.801100 31435 raft_consensus.cc:493] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:00.813072 30674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 282) took 1172 ms. Trace:
W20251025 20:04:00.814393 30748 consensus_meta.cc:287] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.674s user 0.008s sys 0.005s
I20251025 20:04:00.813527 30674 rpcz_store.cc:276] 1025 20:03:59.640703 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.805004 (+164301us) service_pool.cc:225] Handling call
1025 20:03:59.805353 (+ 349us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:59.810358 (+ 5005us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:03:59.810789 (+ 431us) log.cc:844] Serialized 14 byte log entry
1025 20:04:00.146550 (+335761us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:00.798233 (+651683us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b18002ef300. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8278d09a1 00007fd8278dd371 00007fd81ea8dabf 00007fd81ea8fd61 00007fd81ea8fd19 00007fd81ea8fca9 00007fd81ea8fc71 00007fd81ea8ef6d 0000000000379254 0000000000379099 00007fd81ea70d1a
1025 20:04:00.810461 (+ 12228us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b54002966a0. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd826803964 00007fd821ab14fb 00007fd821b3a4cb 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc
1025 20:04:00.810950 (+ 489us) log.cc:844] Serialized 26 byte log entry
1025 20:04:00.811180 (+ 230us) raft_consensus.cc:1645] Marking committed up to 3
1025 20:04:00.811507 (+ 327us) log.cc:844] Serialized 14 byte log entry
1025 20:04:00.811744 (+ 237us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:00.811842 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:00.812825 (+ 983us) raft_consensus.cc:1685] finished
1025 20:04:00.812844 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:00.813002 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4874,"prepare.queue_time_us":199,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":6092,"raft.queue_time_us":15153,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":21011,"spinlock_wait_cycles":96341504,"thread_start_us":272983,"threads_started":1,"wal-append.queue_time_us":698912}
I20251025 20:04:00.814896 30748 raft_consensus.cc:2468] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:00.823765 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:00.805812 31327 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:00.805756 31492 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:00.800967 31378 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:00.885429 30800 raft_consensus.cc:359] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.895244 30800 raft_consensus.cc:385] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:00.800652 30567 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:00.808892 31435 raft_consensus.cc:515] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.895601 30800 raft_consensus.cc:740] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:04:00.885623 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:00.800479 31385 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:00.902905 30745 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:01.146786 30676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 279) took 1597 ms. Trace:
W20251025 20:04:00.898324 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 97819.9us
W20251025 20:04:01.114248 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:01.180634 31423 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:01.113687 31485 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:04:01.258865 31502 log.cc:927] Time spent T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
W20251025 20:04:01.238829 31460 consensus_meta.cc:287] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.521s user 0.013s sys 0.000s
W20251025 20:04:01.181006 31496 log.cc:927] Time spent T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.243s user 0.000s sys 0.000s
W20251025 20:04:01.094969 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 423783us
I20251025 20:04:01.297737 31427 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:01.302002 31414 raft_consensus.cc:3055] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
W20251025 20:04:01.043545 30644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.037s sys 0.078s
I20251025 20:04:00.951562 31488 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:00.895684 31290 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:00.918622 31327 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:01.303951 31455 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:01.305968 31481 raft_consensus.cc:493] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:01.301703 31384 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:01.292390 31385 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:01.313129 30671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 281) took 1731 ms (client timeout 1964 ms). Trace:
I20251025 20:04:01.426744 31327 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:01.426795 30671 rpcz_store.cc:269] 1025 20:03:59.582057 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.586472 (+ 4415us) service_pool.cc:225] Handling call
1025 20:04:01.312676 (+1726204us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b5400169754. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:01.312994 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":929092352}
I20251025 20:04:01.429039 31327 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:01.315256 30596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 374) took 1380 ms. Trace:
W20251025 20:04:01.419118 31499 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.019s sys 0.037s
W20251025 20:04:01.430701 31499 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.679s user 0.019s sys 0.038s
I20251025 20:04:01.419158 31427 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:01.327157 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:01.431973 30752 raft_consensus.cc:2368] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
W20251025 20:04:01.432528 30752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 395) took 1502 ms (client timeout 1780 ms). Trace:
I20251025 20:04:01.105315 30750 raft_consensus.cc:2468] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:01.432875 31427 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:01.433264 30750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 397) took 1494 ms. Trace:
I20251025 20:04:01.433723 30750 rpcz_store.cc:276] 1025 20:03:59.938604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:00.823391 (+884787us) service_pool.cc:225] Handling call
1025 20:04:01.433190 (+609799us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:00.891829 31192 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:01.015343 31446 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.034s sys 0.063s
I20251025 20:04:00.919729 31378 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:01.112669 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 429024us
I20251025 20:04:00.914808 30678 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:01.439276 30678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 294) took 1030 ms. Trace:
I20251025 20:04:01.439848 30678 rpcz_store.cc:276] 1025 20:04:00.409160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:00.668163 (+259003us) service_pool.cc:225] Handling call
1025 20:04:01.439190 (+771027us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:01.348714 31484 log.cc:927] Time spent T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 1.214s user 0.020s sys 0.034s
I20251025 20:04:01.350391 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:01.351540 30793 tablet_bootstrap.cc:492] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:04:01.351547 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:01.355007 30744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 394) took 1426 ms. Trace:
I20251025 20:04:01.041976 30748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 313) took 1108 ms. Trace:
I20251025 20:04:00.917098 31492 raft_consensus.cc:2804] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:01.477108 31492 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:01.477597 31492 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:01.456147 30744 rpcz_store.cc:276] 1025 20:03:59.928315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.930508 (+ 2193us) service_pool.cc:225] Handling call
1025 20:04:01.353808 (+1423300us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:01.354137 (+ 329us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:01.354427 (+ 290us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:01.354473 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:01.354540 (+ 67us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:01.354597 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:01.354687 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:01.354878 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7602944}
I20251025 20:04:01.479802 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 433) took 1553 ms. Trace:
W20251025 20:04:01.367976 31281 consensus_meta.cc:287] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.652s user 0.013s sys 0.000s
I20251025 20:04:01.481508 30747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 391) took 1558 ms. Trace:
I20251025 20:04:01.481716 31281 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:01.482298 30751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 309) took 1713 ms. Trace:
W20251025 20:04:01.446894 31446 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.072s user 0.042s sys 0.070s
I20251025 20:04:01.483759 31281 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:01.482721 30751 rpcz_store.cc:276] 1025 20:03:59.769099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.775472 (+ 6373us) service_pool.cc:225] Handling call
1025 20:03:59.775870 (+ 398us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:59.877364 (+101494us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:59.877472 (+ 108us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:03:59.943457 (+ 65985us) log.cc:844] Serialized 26 byte log entry
1025 20:04:00.751131 (+807674us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:00.751205 (+ 74us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:00.751327 (+ 122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:01.481993 (+730666us) raft_consensus.cc:1685] finished
1025 20:04:01.482023 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:01.482218 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5402,"raft.queue_time_us":20966,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":109,"thread_start_us":859136,"threads_started":2,"wal-append.queue_time_us":768464}
I20251025 20:04:01.375138 30676 rpcz_store.cc:276] 1025 20:03:59.518639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.771606 (+252967us) service_pool.cc:225] Handling call
1025 20:04:01.115965 (+1344359us) spinlock_profiling.cc:243] Waited 36.4 ms on lock 0x7b540007cca8. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd826a47971 00007fd826ac297c 00007fd826af764c 00007fd826ae1efb 00007fd826ae1992 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc
1025 20:04:01.116301 (+ 336us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":83608832}
I20251025 20:04:00.948808 30800 consensus_queue.cc:260] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:01.379347 30745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 318) took 1211 ms. Trace:
I20251025 20:04:01.423935 30719 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:04:01.486439 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 62625.3us
I20251025 20:04:01.424927 31481 raft_consensus.cc:515] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:01.425925 31384 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:01.437413 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:01.513286 30750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 403) took 1146 ms. Trace:
I20251025 20:04:01.513723 30750 rpcz_store.cc:276] 1025 20:04:00.366485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:01.437032 (+1070547us) service_pool.cc:225] Handling call
1025 20:04:01.513194 (+ 76162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:01.429814 30596 rpcz_store.cc:276] 1025 20:03:59.934487 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.938212 (+ 3725us) service_pool.cc:225] Handling call
1025 20:04:01.314071 (+1375859us) spinlock_profiling.cc:243] Waited 7.01 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:01.314398 (+ 327us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:01.314686 (+ 288us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:01.314734 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:01.314806 (+ 72us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:01.314857 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:01.314960 (+ 103us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:01.315140 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16111488}
W20251025 20:04:01.432962 30752 rpcz_store.cc:269] 1025 20:03:59.930419 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.930574 (+ 155us) service_pool.cc:225] Handling call
1025 20:04:01.326901 (+1396327us) spinlock_profiling.cc:243] Waited 6.32 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:01.432427 (+105526us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14527872}
W20251025 20:04:01.419665 30644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.392s user 0.043s sys 0.083s
I20251025 20:04:01.429636 31327 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:01.437474 31290 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:01.441551 30673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.013s sys 0.020s
I20251025 20:04:01.416832 31414 raft_consensus.cc:740] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:01.429647 30671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 295) took 1019 ms. Trace:
I20251025 20:04:01.405045 31485 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:01.397377 31423 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:01.442024 31435 leader_election.cc:290] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:01.417816 31460 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:01.485623 30745 rpcz_store.cc:276] 1025 20:04:00.208430 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:00.391634 (+183204us) service_pool.cc:225] Handling call
1025 20:04:01.379259 (+987625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:01.512207 31446 raft_consensus.cc:2749] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:04:01.446784 30675 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.003s sys 0.032s
I20251025 20:04:01.568099 30800 ts_tablet_manager.cc:1434] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.848s user 0.018s sys 0.042s
I20251025 20:04:01.435088 31488 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:01.510918 31473 log.cc:927] Time spent T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 1.513s user 0.006s sys 0.015s
I20251025 20:04:01.589426 31435 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:01.667233 31435 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:01.480388 30677 rpcz_store.cc:276] 1025 20:03:59.926119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:00.202490 (+276371us) service_pool.cc:225] Handling call
1025 20:04:00.240725 (+ 38235us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:00.241286 (+ 561us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:00.241335 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:00.378370 (+137035us) log.cc:844] Serialized 26 byte log entry
1025 20:04:00.755427 (+377057us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:00.755494 (+ 67us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:00.755638 (+ 144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:01.479473 (+723835us) raft_consensus.cc:1685] finished
1025 20:04:01.479515 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:01.479717 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8427,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":39133,"thread_start_us":332658,"threads_started":1,"wal-append.queue_time_us":639314}
I20251025 20:04:01.671567 31460 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:01.672014 31290 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:01.591310 31423 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:01.437696 31455 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:01.481976 30747 rpcz_store.cc:276] 1025 20:03:59.923048 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.924825 (+ 1777us) service_pool.cc:225] Handling call
1025 20:04:01.350212 (+1425387us) spinlock_profiling.cc:243] Waited 8.18 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea507ce 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:01.481431 (+131219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18825472}
I20251025 20:04:01.447584 31192 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:01.586822 30673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s user 0.013s sys 0.021s
W20251025 20:04:01.591841 30749 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.036s sys 0.048s
W20251025 20:04:01.620522 30800 ts_tablet_manager.cc:1467] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957: Tablet startup took 900ms
I20251025 20:04:01.699666 31414 consensus_queue.cc:260] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:01.699687 30749 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.815s user 0.036s sys 0.048s
W20251025 20:04:01.699764 30800 ts_tablet_manager.cc:1469] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957: Trace:
1025 20:03:40.519492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.609974 (+1090482us) service_pool.cc:225] Handling call
1025 20:03:41.622232 (+ 12258us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.622369 (+ 137us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.709079 (+ 86710us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.724397 (+ 15318us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.732224 (+ 7827us) inbound_call.cc:173] Queueing success response
1025 20:03:59.328330 (+17596106us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1025 20:04:00.720228 (+1391898us) ts_tablet_manager.cc:1435] Starting tablet replica
1025 20:04:00.807892 (+ 87664us) tablet_replica.cc:261] Starting instrumentation
1025 20:04:00.808561 (+ 669us) tablet_replica.cc:280] Starting consensus
Metrics: {"tablet-open.queue_time_us":17598319}
I20251025 20:04:01.703042 30679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 430) took 1838 ms. Trace:
I20251025 20:04:01.703593 30679 rpcz_store.cc:276] 1025 20:03:59.865074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.945927 (+ 80853us) service_pool.cc:225] Handling call
1025 20:04:01.701944 (+1756017us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:01.702277 (+ 333us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:01.702541 (+ 264us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:01.702587 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:01.702649 (+ 62us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:01.702707 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:01.702790 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:01.702953 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9487744}
I20251025 20:04:01.575974 30671 rpcz_store.cc:276] 1025 20:04:00.409842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:01.428408 (+1018566us) service_pool.cc:225] Handling call
1025 20:04:01.428894 (+ 486us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:01.429149 (+ 255us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:01.429194 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:01.429260 (+ 66us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:01.429308 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:01.429399 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:01.429570 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:01.591986 30672 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.854s user 0.043s sys 0.075s
W20251025 20:04:01.621619 30675 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.755s user 0.004s sys 0.032s
W20251025 20:04:01.634236 31478 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.008s sys 0.037s
I20251025 20:04:01.731702 30597 raft_consensus.cc:2393] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
W20251025 20:04:01.748684 30570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.008s sys 0.026s
W20251025 20:04:01.749177 30570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.800s user 0.008s sys 0.026s
I20251025 20:04:01.567314 31327 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
W20251025 20:04:01.792587 30674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.975s user 0.034s sys 0.105s
W20251025 20:04:01.907119 30796 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.003s sys 0.012s
W20251025 20:04:01.953483 30674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.136s user 0.044s sys 0.119s
W20251025 20:04:01.922747 30645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.935s user 0.005s sys 0.010s
W20251025 20:04:01.954245 30645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.966s user 0.006s sys 0.010s
W20251025 20:04:01.957615 30796 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.654s user 0.003s sys 0.013s
W20251025 20:04:01.748965 30672 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.992s user 0.043s sys 0.075s
W20251025 20:04:01.979218 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20251025 20:04:01.754174 31478 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.997s user 0.008s sys 0.038s
I20251025 20:04:01.472960 30748 rpcz_store.cc:276] 1025 20:03:59.934000 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:00.815850 (+881850us) service_pool.cc:225] Handling call
1025 20:04:01.040988 (+225138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:01.667274 31481 leader_election.cc:290] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:02.009361 30567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.006s sys 0.018s
I20251025 20:04:01.748533 31503 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:02.018038 30569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.000s sys 0.018s
W20251025 20:04:02.018697 30569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.984s user 0.000s sys 0.018s
I20251025 20:04:02.018493 31503 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:02.019119 30568 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.774s user 0.016s sys 0.010s
W20251025 20:04:02.019445 30568 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.927s user 0.016s sys 0.010s
W20251025 20:04:01.900116 30642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.006s sys 0.007s
I20251025 20:04:01.759603 31290 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:01.884840 30746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.003s sys 0.011s
I20251025 20:04:01.955644 31414 raft_consensus.cc:3060] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:02.017773 30748 raft_consensus.cc:1275] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251025 20:04:01.873275 31413 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.982s user 0.010s sys 0.006s
W20251025 20:04:02.029865 30642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.989s user 0.008s sys 0.007s
I20251025 20:04:02.002470 31318 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:02.012001 30567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s user 0.006s sys 0.018s
W20251025 20:04:02.061769 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 686199us
I20251025 20:04:02.062920 30567 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:01.758675 31460 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:01.822781 31488 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:02.042114 30753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 306) took 2269 ms. Trace:
W20251025 20:04:01.952603 31234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.008s sys 0.008s
I20251025 20:04:02.018493 30800 tablet_bootstrap.cc:492] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
W20251025 20:04:02.039652 30746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s user 0.003s sys 0.013s
I20251025 20:04:02.036147 31481 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:02.041782 31378 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:02.043553 31413 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.158s user 0.012s sys 0.008s
W20251025 20:04:02.084704 31234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.734s user 0.011s sys 0.008s
I20251025 20:04:02.042599 31290 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.074589 31318 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.065127 31506 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:02.090864 31518 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
W20251025 20:04:02.045243 31492 consensus_meta.cc:287] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.565s user 0.007s sys 0.004s
I20251025 20:04:02.043474 31495 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:02.043913 31455 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:02.061067 31385 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:02.560230 30674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 293) took 2152 ms. Trace:
I20251025 20:04:02.505959 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:02.623117 30674 rpcz_store.cc:276] 1025 20:04:00.408145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:00.815513 (+407368us) service_pool.cc:225] Handling call
1025 20:04:00.815938 (+ 425us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:00.816329 (+ 391us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:00.816356 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:00.816837 (+ 481us) log.cc:844] Serialized 26 byte log entry
1025 20:04:01.954200 (+1137363us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:04:01.954248 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:01.954322 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:02.559842 (+605520us) raft_consensus.cc:1685] finished
1025 20:04:02.559877 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:02.560125 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":663,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":1136866,"threads_started":1,"wal-append.queue_time_us":1645163}
I20251025 20:04:02.288658 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:02.625372 30750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 325) took 1203 ms. Trace:
W20251025 20:04:02.563045 30643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.127s user 0.026s sys 0.024s
W20251025 20:04:02.626504 30643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.657s user 0.028s sys 0.025s
W20251025 20:04:02.206856 30597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 378) took 2270 ms (client timeout 1970 ms). Trace:
I20251025 20:04:02.357795 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:02.630959 30745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 326) took 1156 ms. Trace:
I20251025 20:04:02.631428 30745 rpcz_store.cc:276] 1025 20:04:01.474571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.357324 (+882753us) service_pool.cc:225] Handling call
1025 20:04:02.630866 (+273542us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:02.065840 30753 rpcz_store.cc:276] 1025 20:03:59.768073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.769991 (+ 1918us) service_pool.cc:225] Handling call
1025 20:03:59.770340 (+ 349us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:03:59.770698 (+ 358us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:03:59.770733 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:03:59.771211 (+ 478us) log.cc:844] Serialized 26 byte log entry
1025 20:03:59.904227 (+133016us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:03:59.904269 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:03:59.905707 (+ 1438us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:01.347872 (+1442165us) spinlock_profiling.cc:243] Waited 95.2 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:02.036813 (+688941us) raft_consensus.cc:1685] finished
1025 20:04:02.036847 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:02.037101 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14398,"raft.queue_time_us":2666,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6241,"spinlock_wait_cycles":218947328,"thread_start_us":132575,"threads_started":1,"wal-append.queue_time_us":292123}
I20251025 20:04:02.073690 30748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 314) took 2059 ms. Trace:
I20251025 20:04:02.639505 30752 raft_consensus.cc:2393] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
W20251025 20:04:02.573017 30567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.002s sys 0.007s
W20251025 20:04:02.663393 30567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s user 0.003s sys 0.008s
W20251025 20:04:02.351765 30749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.021s sys 0.020s
I20251025 20:04:02.662966 30752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 407) took 1429 ms. Trace:
W20251025 20:04:02.526111 31522 log.cc:927] Time spent T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
W20251025 20:04:02.261498 31431 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.007s sys 0.030s
I20251025 20:04:02.351312 31497 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:02.572697 30670 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.212s user 0.005s sys 0.005s
W20251025 20:04:02.665526 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 602666us
I20251025 20:04:02.635317 30748 rpcz_store.cc:276] 1025 20:04:00.005296 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.016917 (+2011621us) service_pool.cc:225] Handling call
1025 20:04:02.017326 (+ 409us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:02.043318 (+ 25992us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:02.064916 (+ 21598us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:02.290800 31499 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.019s sys 0.029s
I20251025 20:04:02.519438 30744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 396) took 2582 ms. Trace:
I20251025 20:04:02.085446 31488 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:02.482406 31454 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:02.520881 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:02.176483 31485 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:02.225203 31513 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:02.615159 31446 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.002s sys 0.027s
I20251025 20:04:02.670128 30752 rpcz_store.cc:276] 1025 20:04:01.234126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.482712 (+1248586us) service_pool.cc:225] Handling call
1025 20:04:02.662883 (+180171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:02.672183 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:02.839069 31440 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:02.669776 31491 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:02.680006 30793 tablet_bootstrap.cc:654] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:02.261157 30675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 290) took 1855 ms. Trace:
I20251025 20:04:02.525823 30673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 431) took 2657 ms. Trace:
I20251025 20:04:02.694499 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:02.635797 31503 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:02.670176 30570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.009s sys 0.059s
W20251025 20:04:02.862920 30570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.037s user 0.009s sys 0.060s
I20251025 20:04:02.180174 31435 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:02.674602 30500 catalog_manager.cc:5649] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
W20251025 20:04:02.637501 30597 rpcz_store.cc:269] 1025 20:03:59.936412 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.936579 (+ 167us) service_pool.cc:225] Handling call
1025 20:04:01.351331 (+1414752us) spinlock_profiling.cc:243] Waited 4.95 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:02.206438 (+855107us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54002ff654. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:02.206699 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":666020864}
I20251025 20:04:02.073666 31460 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.661926 31327 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:02.640677 31192 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:02.674875 31431 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.249s user 0.007s sys 0.056s
W20251025 20:04:02.669401 30749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.968s user 0.028s sys 0.039s
I20251025 20:04:02.669142 31504 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:02.636021 31492 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:02.818343 30568 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.798s user 0.011s sys 0.005s
W20251025 20:04:02.976719 30568 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.957s user 0.011s sys 0.005s
I20251025 20:04:02.116775 31481 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.645558 31385 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.517493 31391 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:02.646315 31455 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:02.456130 31524 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:02.664192 31505 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:02.625828 30750 rpcz_store.cc:276] 1025 20:04:01.422133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.288211 (+866078us) service_pool.cc:225] Handling call
1025 20:04:02.625271 (+337060us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:02.661700 31378 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:02.838279 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 177107us
I20251025 20:04:02.485471 31462 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:02.579286 31517 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:02.664189 31525 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:02.361155 31508 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:02.645807 31506 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.662179 31234 raft_consensus.cc:2749] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:04:02.646006 31518 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.582788 31428 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:02.777105 31384 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:02.845768 30567 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:02.648492 31495 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.839371 31501 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
W20251025 20:04:02.846244 30670 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.483s user 0.008s sys 0.007s
W20251025 20:04:02.847637 31499 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.416s user 0.021s sys 0.057s
W20251025 20:04:02.848569 31461 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.555s user 0.019s sys 0.014s
I20251025 20:04:02.669188 31281 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:02.841154 31493 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:02.860378 30753 raft_consensus.cc:2410] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251025 20:04:02.861061 30645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.842s user 0.008s sys 0.008s
W20251025 20:04:03.047973 30645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.029s user 0.009s sys 0.009s
W20251025 20:04:02.848749 31478 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.791s user 0.002s sys 0.014s
W20251025 20:04:02.857717 31446 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.184s user 0.002s sys 0.027s
I20251025 20:04:02.838946 31535 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:02.862226 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:02.842444 31514 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:02.843662 31423 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:04:02.853403 30642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.035s sys 0.124s
I20251025 20:04:02.903564 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:02.904031 30500 catalog_manager.cc:5649] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 2 to 3, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:02.976425 30747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 408) took 1702 ms. Trace:
I20251025 20:04:02.902010 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:02.996562 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:04:02.983063 31414 consensus_meta.cc:287] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.938s user 0.013s sys 0.000s
I20251025 20:04:02.910917 30746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 393) took 2978 ms. Trace:
I20251025 20:04:03.090375 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:02.978215 31192 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:02.854377 31488 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.854149 31523 consensus_queue.cc:1048] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.151s
I20251025 20:04:02.979174 31262 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:02.854066 31454 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.982553 31327 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:03.107489 31327 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.105752 31413 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:02.982424 31435 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:03.109758 31413 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:03.110927 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:03.110456 31435 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.901362 31485 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:02.989564 31318 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.054702 30796 tablet_bootstrap.cc:492] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:04:03.135386 30796 ts_tablet_manager.cc:1403] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 3.107s user 0.016s sys 0.052s
W20251025 20:04:03.065088 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 219403us
I20251025 20:04:03.052459 31519 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:03.065709 30753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 328) took 1048 ms. Trace:
W20251025 20:04:03.052382 31478 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.008s user 0.002s sys 0.015s
I20251025 20:04:03.071687 30745 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:03.139463 30745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 327) took 1122 ms. Trace:
I20251025 20:04:03.139947 30745 rpcz_store.cc:276] 1025 20:04:02.017094 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.857415 (+840321us) service_pool.cc:225] Handling call
1025 20:04:03.139373 (+281958us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:02.994346 30749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 390) took 3071 ms. Trace:
I20251025 20:04:02.848201 30744 rpcz_store.cc:276] 1025 20:03:59.937022 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.518036 (+2581014us) service_pool.cc:225] Handling call
1025 20:04:02.518589 (+ 553us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:02.518881 (+ 292us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:02.518925 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:02.519022 (+ 97us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:02.519073 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:02.519162 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:02.519349 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:02.859560 31497 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.032214 31530 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:02.989626 31455 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.049871 31234 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:03.091593 31384 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:02.982973 31504 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.999565 31525 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.866488 31513 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.091022 30500 catalog_manager.cc:5649] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 1, leader changed from <none> to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 1 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:03.089573 31423 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.054210 31290 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:03.092947 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:04:03.002138 31508 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.999130 31378 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:03.084916 30642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.996s user 0.035s sys 0.124s
I20251025 20:04:03.087572 31503 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:03.052204 31281 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.078174 31537 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:03.052234 31462 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.864610 31491 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.138628 30753 rpcz_store.cc:276] 1025 20:04:02.017499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.661042 (+643543us) service_pool.cc:225] Handling call
1025 20:04:03.065616 (+404574us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:02.995906 31505 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.986125 31391 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.099177 30748 raft_consensus.cc:2468] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:04:03.104012 31414 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:03.169548 31413 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:03.089151 30747 rpcz_store.cc:276] 1025 20:04:01.274457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.520432 (+1245975us) service_pool.cc:225] Handling call
1025 20:04:02.910660 (+390228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:02.865288 31440 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.049626 31501 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.114393 30670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 429) took 3257 ms. Trace:
I20251025 20:04:03.200871 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:03.200574 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:02.999758 31517 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.106110 31531 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:03.066967 31493 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.097404 30752 raft_consensus.cc:2468] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
W20251025 20:04:03.215853 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 79094.5us
W20251025 20:04:03.170823 30716 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.192.194:32975 timed out after 1.727s (SENT) [suppressed 3 similar messages]
I20251025 20:04:02.993309 31524 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.204784 30500 catalog_manager.cc:5649] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:03.083595 31535 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.103905 31192 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.227810 30750 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:03.225865 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:03.134644 31262 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.164906 31532 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:03.138964 31485 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.174436 31209 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:03.134158 30596 raft_consensus.cc:1275] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:03.200585 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:03.238850 30603 raft_consensus.cc:1240] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:03.154206 31519 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.175324 31318 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:03.252854 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:03.221248 31478 raft_consensus.cc:493] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:03.259191 31545 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:03.221163 31290 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:03.205387 31541 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:03.293851 30645 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:03.309239 30753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 315) took 3285 ms. Trace:
I20251025 20:04:03.399683 30753 rpcz_store.cc:276] 1025 20:04:00.023646 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.233257 (+3209611us) service_pool.cc:225] Handling call
1025 20:04:03.308595 (+ 75338us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:03.308832 (+ 237us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:03.308866 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:03.308919 (+ 53us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:03.308954 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:03.309034 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:03.309167 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:03.241009 31492 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.237596 31460 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.169432 31234 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.171510 31384 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.232067 31435 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.315822 30597 raft_consensus.cc:2410] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:03.239745 31327 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:03.361303 31527 log.cc:927] Time spent T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.372s user 0.000s sys 0.001s
I20251025 20:04:03.260378 30748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 330) took 1191 ms. Trace:
I20251025 20:04:03.363855 30602 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:03.085858 31514 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.350997 30500 catalog_manager.cc:5649] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 0 to 2, leader changed from <none> to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 2 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:03.456562 30500 catalog_manager.cc:5649] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:03.458632 30500 catalog_manager.cc:5649] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:03.307166 30599 raft_consensus.cc:1275] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:03.366752 31431 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:03.372061 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:03.090803 30746 rpcz_store.cc:276] 1025 20:03:59.925111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.927140 (+ 2029us) service_pool.cc:225] Handling call
1025 20:04:01.298737 (+1371597us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:01.299038 (+ 301us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:01.299575 (+ 537us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:01.299622 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:01.321728 (+ 22106us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b2c000bb0d8. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8204399b1 00007fd820448fe2 00007fd821b75701 00007fd821b39fda 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22
1025 20:04:01.322587 (+ 859us) log.cc:844] Serialized 26 byte log entry
1025 20:04:02.088755 (+766168us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:02.088813 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:02.088907 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:02.857599 (+768692us) raft_consensus.cc:1685] finished
1025 20:04:02.857630 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:02.903673 (+ 46043us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3950,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":144,"spinlock_wait_cycles":56286976,"thread_start_us":761436,"threads_started":1,"wal-append.queue_time_us":901875}
I20251025 20:04:03.355077 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:03.373668 31535 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.314594 31523 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:03.308836 31385 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:03.381255 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 60962.4us
I20251025 20:04:03.460898 30500 catalog_manager.cc:5649] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 2, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:03.237429 31503 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.261855 31414 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.320530 31531 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:02.861977 30675 rpcz_store.cc:276] 1025 20:04:00.405775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:00.411765 (+ 5990us) service_pool.cc:225] Handling call
1025 20:04:00.634550 (+222785us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:00.696958 (+ 62408us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:00.701182 (+ 4224us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:00.757884 (+ 56702us) log.cc:844] Serialized 26 byte log entry
1025 20:04:01.852463 (+1094579us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:01.865600 (+ 13137us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:01.957821 (+ 92221us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:02.207179 (+249358us) raft_consensus.cc:1685] finished
1025 20:04:02.224224 (+ 17045us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:02.261035 (+ 36811us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":65154,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":11259,"thread_start_us":919440,"threads_started":1,"wal-append.queue_time_us":1403768}
I20251025 20:04:03.314587 31548 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:03.357438 30570 connection.cc:232] RPC call timeout handler was delayed by -1.5503658180000457s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
W20251025 20:04:03.403815 31534 log.cc:927] Time spent T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.329s user 0.001s sys 0.000s
I20251025 20:04:03.404520 31290 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.405105 31460 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:03.317873 31481 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.419437 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:03.420256 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:03.378804 30744 raft_consensus.cc:3060] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:03.364787 31418 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:03.369808 31461 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:03.236043 31530 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.237401 31537 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.452477 31545 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.373659 31493 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:03.378512 31506 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:03.359417 31532 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.497792 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:02.862133 30673 rpcz_store.cc:276] 1025 20:03:59.868554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:00.005651 (+137097us) service_pool.cc:225] Handling call
1025 20:04:00.077730 (+ 72079us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:00.331344 (+253614us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:00.337139 (+ 5795us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:00.805129 (+467990us) spinlock_profiling.cc:243] Waited 9.29 ms on lock 0x7b2c00010828. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8204399b1 00007fd820448fe2 00007fd821b75701 00007fd821b39fda 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22
1025 20:04:00.805631 (+ 502us) log.cc:844] Serialized 26 byte log entry
1025 20:04:01.778021 (+972390us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:01.855997 (+ 77976us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:01.874762 (+ 18765us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:02.525331 (+650569us) raft_consensus.cc:1685] finished
1025 20:04:02.525372 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:02.525720 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24808,"raft.queue_time_us":38178,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":58048,"spinlock_wait_cycles":21375360,"thread_start_us":918027,"threads_started":1,"wal-append.queue_time_us":1439464}
I20251025 20:04:03.361945 31209 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.396216 31478 raft_consensus.cc:515] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.551215 30752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 331) took 1486 ms. Trace:
I20251025 20:04:03.388123 31318 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.414618 31541 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.446596 30748 rpcz_store.cc:276] 1025 20:04:02.060393 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.852592 (+792199us) service_pool.cc:225] Handling call
1025 20:04:03.251618 (+399026us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:03.551371 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:03.570577 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 261950us
W20251025 20:04:03.597442 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:04.003286 30796 raft_consensus.cc:359] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.004369 30796 raft_consensus.cc:385] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:04.004858 30796 raft_consensus.cc:740] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:04:04.005764 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:03.550194 30500 catalog_manager.cc:5649] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:04.006603 30752 rpcz_store.cc:276] 1025 20:04:02.064890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.856954 (+792064us) service_pool.cc:225] Handling call
1025 20:04:03.550974 (+694020us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b54002d7654. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:03.551122 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27063168}
I20251025 20:04:03.496599 31435 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:03.467878 31327 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:03.498185 31234 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:04.011046 30570 connection.cc:232] RPC call timeout handler was delayed by -1.5452981270000237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2036s for a response
I20251025 20:04:04.011512 31543 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:04.011482 31552 raft_consensus.cc:2804] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:04:03.623795 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:04.025234 30674 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:04:03.823469 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 652613us
W20251025 20:04:04.027762 30570 connection.cc:232] RPC call timeout handler was delayed by -1.002587819999917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1685s for a response
W20251025 20:04:04.021529 30645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.000s sys 0.016s
W20251025 20:04:04.053581 30645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.640s user 0.000s sys 0.016s
I20251025 20:04:03.470757 31518 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.962482 31391 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.548923 30604 raft_consensus.cc:2468] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
W20251025 20:04:03.999542 31553 log.cc:927] Time spent T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.229s user 0.000s sys 0.000s
I20251025 20:04:04.010067 30500 catalog_manager.cc:5649] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:04.084772 30500 catalog_manager.cc:5649] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:04.071768 30676 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:04.070119 31555 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:04.072527 30753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 320) took 3693 ms. Trace:
W20251025 20:04:04.073064 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 779305us
I20251025 20:04:04.001104 30671 raft_consensus.cc:2468] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:03.938131 31423 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:03.938282 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:03.671643 31262 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:03.745198 31384 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.904022 31281 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.019245 31460 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.590088 31454 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.714748 31542 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:03.746423 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.016265 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.077184 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:03.715736 30747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 319) took 3337 ms. Trace:
I20251025 20:04:04.131246 30750 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:03.749444 31513 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.008726 31192 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.734138 31547 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:03.852463 31488 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.017962 30796 consensus_queue.cc:260] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.520530 31431 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.082127 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:04.135594 30674 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 5: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:03.964715 31554 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:04.088660 30753 rpcz_store.cc:276] 1025 20:04:00.379127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.071522 (+3692395us) service_pool.cc:225] Handling call
1025 20:04:04.071927 (+ 405us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.072141 (+ 214us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.072182 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.072227 (+ 45us) raft_consensus.cc:1645] Marking committed up to 3
1025 20:04:04.072263 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.072322 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.072464 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.090307 30645 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:04.137048 30750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 405) took 3756 ms. Trace:
I20251025 20:04:04.091851 30500 catalog_manager.cc:5649] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:04.138855 30500 catalog_manager.cc:5649] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:04.074502 31481 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:04.077816 31234 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:04.014218 31418 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:04.083079 31446 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.988s user 0.008s sys 0.014s
I20251025 20:04:04.015446 31461 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:04.129858 31556 raft_consensus.cc:493] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:04.141096 30500 catalog_manager.cc:5649] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:04.155045 30500 catalog_manager.cc:5649] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 2, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 2 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:03.145119 30749 rpcz_store.cc:276] 1025 20:03:59.922578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.924759 (+ 2181us) service_pool.cc:225] Handling call
1025 20:04:00.812361 (+887602us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:00.812610 (+ 249us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:00.812851 (+ 241us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:01.701567 (+888716us) log.cc:844] Serialized 14 byte log entry
1025 20:04:02.979063 (+1277496us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:02.982071 (+ 3008us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:02.984451 (+ 2380us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:02.984540 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:02.991903 (+ 7363us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71759,"prepare.queue_time_us":888862,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":60,"raft.queue_time_us":102324,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":6935,"spinlock_wait_cycles":7463296,"thread_start_us":2025402,"threads_started":2,"wal-append.queue_time_us":879195}
I20251025 20:04:04.077840 31501 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.119292 31557 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:03.626458 31523 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.083719 31391 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:04.026867 31327 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.090634 31503 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.088650 31493 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:03.935371 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:04.132581 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 58541.3us
I20251025 20:04:04.188481 30570 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:04.134586 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 54372.6us
I20251025 20:04:04.190052 30796 ts_tablet_manager.cc:1434] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 1.054s user 0.016s sys 0.003s
W20251025 20:04:04.190598 30796 ts_tablet_manager.cc:1467] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d: Tablet startup took 1054ms
I20251025 20:04:04.108973 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:04.114377 30675 raft_consensus.cc:3060] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 4
I20251025 20:04:04.084312 31543 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.112112 31423 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:04:04.132288 31540 log.cc:927] Time spent T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.394s user 0.001s sys 0.000s
I20251025 20:04:04.137431 30750 rpcz_store.cc:276] 1025 20:04:00.380710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.136122 (+3755412us) service_pool.cc:225] Handling call
1025 20:04:04.136497 (+ 375us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.136691 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.136719 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.136765 (+ 46us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.136797 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.136853 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.136977 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:03.903437 31525 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:03.550035 31495 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:03.747201 31505 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.024063 31435 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.129513 31462 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.079279 31518 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:04.019498 31548 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:03.748534 31508 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.106868 31545 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.027745 31504 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:04.145985 31446 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.055s user 0.009s sys 0.015s
I20251025 20:04:04.084427 31552 raft_consensus.cc:697] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
W20251025 20:04:04.167404 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 77190us
I20251025 20:04:04.104138 31537 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.135308 31492 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:03.750300 31440 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.167968 30676 raft_consensus.cc:2468] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:04.085490 31497 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.086977 31514 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:04.154354 31550 log.cc:927] Time spent T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.495s user 0.001s sys 0.000s
I20251025 20:04:04.252576 30753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 321) took 3870 ms. Trace:
I20251025 20:04:04.146443 31414 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.169272 30500 catalog_manager.cc:5649] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 3 to 4, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:04.196692 31493 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.195261 31290 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.195851 30746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 323) took 3799 ms. Trace:
W20251025 20:04:04.265966 31565 log.cc:927] Time spent T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20251025 20:04:04.266150 30746 rpcz_store.cc:276] 1025 20:04:00.397723 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.181554 (+3783831us) service_pool.cc:225] Handling call
1025 20:04:04.195185 (+ 13631us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.195427 (+ 242us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.195462 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.195514 (+ 52us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.195579 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.195650 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.195782 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.258195 31492 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.268599 30746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 324) took 2891 ms. Trace:
I20251025 20:04:04.269150 31492 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.225083 31563 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:04.158449 30748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 404) took 3791 ms. Trace:
I20251025 20:04:04.166953 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:04.182237 31546 log.cc:927] Time spent T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.545s user 0.001s sys 0.000s
I20251025 20:04:04.205943 31478 leader_election.cc:290] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.113209 31555 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.151425 31524 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.168989 31547 raft_consensus.cc:697] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:04.274963 30604 raft_consensus.cc:2393] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:04.270318 31563 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.178371 31281 raft_consensus.cc:2804] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election won for term 3
W20251025 20:04:04.190874 30796 ts_tablet_manager.cc:1469] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d: Trace:
1025 20:03:40.252222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:41.258888 (+1006666us) service_pool.cc:225] Handling call
1025 20:03:41.346246 (+ 87358us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:03:41.346350 (+ 104us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:03:41.539512 (+193162us) tablet_metadata.cc:641] Metadata flushed
1025 20:03:41.547886 (+ 8374us) tablet_replica.cc:206] Creating consensus instance
1025 20:03:41.548286 (+ 400us) inbound_call.cc:173] Queueing success response
1025 20:04:00.019399 (+18471113us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1025 20:04:03.135757 (+3116358us) ts_tablet_manager.cc:1435] Starting tablet replica
1025 20:04:03.746884 (+611127us) tablet_replica.cc:261] Starting instrumentation
1025 20:04:03.747613 (+ 729us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":16648,"raft.queue_time_us":32623,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":9397,"tablet-open.queue_time_us":18471173}
I20251025 20:04:03.254621 30670 rpcz_store.cc:276] 1025 20:03:59.857128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.926837 (+ 69709us) service_pool.cc:225] Handling call
1025 20:04:01.358995 (+1432158us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:01.359315 (+ 320us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:01.359921 (+ 606us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:01.359967 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:01.360697 (+ 730us) log.cc:844] Serialized 26 byte log entry
1025 20:04:02.260026 (+899329us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd81ea7059c 00007fd827849219 00007fd81ea8cc4b 00007fd81ea8aeef 00007fd81ea8d1b2 00007fd8216d42bb
1025 20:04:03.075367 (+815341us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:03.077920 (+ 2553us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:03.089738 (+ 11818us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:03.090914 (+ 1176us) raft_consensus.cc:1685] finished
1025 20:04:03.096333 (+ 5419us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:03.106709 (+ 10376us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1645,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":133,"spinlock_wait_cycles":40577152,"thread_start_us":1690057,"threads_started":1,"wal-append.queue_time_us":1221925,"wal-append.run_cpu_time_us":1789,"wal-append.run_wall_time_us":1001808}
I20251025 20:04:04.168107 31491 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.274070 30671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 441) took 1042 ms. Trace:
I20251025 20:04:04.216080 30750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 406) took 3042 ms. Trace:
I20251025 20:04:04.151648 31481 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.148685 31234 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.177477 31519 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.198750 31488 raft_consensus.cc:2749] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:04.283627 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:04.238657 31505 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:04.167721 31559 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:04.131696 30747 rpcz_store.cc:276] 1025 20:04:00.334594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.315613 (+2981019us) service_pool.cc:225] Handling call
1025 20:04:03.407460 (+ 91847us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:03.468505 (+ 61045us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:03.520018 (+ 51513us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:03.535501 (+ 15483us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:03.549082 (+ 13581us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:03.581273 (+ 32191us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:03.671445 (+ 90172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.241858 31543 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:04.243134 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 53160.2us
I20251025 20:04:04.245321 30645 leader_election.cc:400] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:04.195891 31532 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.247485 30719 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:04.144011 31542 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.210029 30673 raft_consensus.cc:2468] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:04.187999 31391 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:04.253217 30602 consensus_meta.cc:287] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.804s user 0.009s sys 0.003s
I20251025 20:04:04.206913 31570 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:04.096050 31517 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.226770 31318 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.258394 30753 rpcz_store.cc:276] 1025 20:04:00.382564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.137449 (+3754885us) service_pool.cc:225] Handling call
1025 20:04:04.137753 (+ 304us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.138096 (+ 343us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:04.213043 (+ 74947us) log.cc:844] Serialized 14 byte log entry
1025 20:04:04.252208 (+ 39165us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.252265 (+ 57us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:04.252308 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.252379 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.252511 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12573,"prepare.queue_time_us":67742,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":12736,"thread_start_us":84776,"threads_started":2}
I20251025 20:04:04.157833 31554 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.255170 31385 consensus_queue.cc:1048] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20251025 20:04:04.270648 30679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 436) took 1043 ms. Trace:
I20251025 20:04:04.327126 30679 rpcz_store.cc:276] 1025 20:04:03.227348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.229993 (+ 2645us) service_pool.cc:225] Handling call
1025 20:04:03.362102 (+132109us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.081796 (+719694us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.083626 (+ 1830us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.090654 (+ 7028us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.094697 (+ 4043us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.270348 (+175651us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b54000a73d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:04.270406 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.270580 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1637357824}
I20251025 20:04:04.209750 31423 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.269030 30746 rpcz_store.cc:276] 1025 20:04:01.377612 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.267456 (+2889844us) service_pool.cc:225] Handling call
1025 20:04:04.267915 (+ 459us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.268160 (+ 245us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.268197 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.268257 (+ 60us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.268298 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.268374 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.268530 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.244642 31549 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:04.271903 31460 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.258925 31209 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.212309 31557 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.283113 30750 rpcz_store.cc:276] 1025 20:04:01.174159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.203669 (+3029510us) service_pool.cc:225] Handling call
1025 20:04:04.215260 (+ 11591us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.215569 (+ 309us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.215613 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.215684 (+ 71us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.215732 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.215814 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.215978 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:04.285368 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:04:04.240329 31518 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:04.288174 31536 log.cc:927] Time spent T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20251025 20:04:04.253625 31504 consensus_queue.cc:1059] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251025 20:04:04.238476 31562 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:04.271592 31530 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.283311 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:04.303756 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 58507.7us
W20251025 20:04:04.358368 30645 leader_election.cc:343] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:04:04.358963 30645 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:04.360658 30645 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:04.307507 30796 tablet_bootstrap.cc:492] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:04:04.172191 31556 raft_consensus.cc:515] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:04.311234 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 63846.3us
I20251025 20:04:04.315848 30602 raft_consensus.cc:2468] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:04.320122 31531 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.324831 31523 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.325646 30570 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:04.325754 30753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 410) took 1659 ms. Trace:
I20251025 20:04:04.367506 30753 rpcz_store.cc:276] 1025 20:04:02.666039 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.324382 (+1658343us) service_pool.cc:225] Handling call
1025 20:04:04.324923 (+ 541us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.325430 (+ 507us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.325667 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.289156 31281 raft_consensus.cc:697] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:04.338925 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:04.304509 31492 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:04.272509 30748 rpcz_store.cc:276] 1025 20:04:00.366841 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.074217 (+3707376us) service_pool.cc:225] Handling call
1025 20:04:04.096017 (+ 21800us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.127311 (+ 31294us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.129812 (+ 2501us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.132300 (+ 2488us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.135896 (+ 3596us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.144961 (+ 9065us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.154453 (+ 9492us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.318625 31543 raft_consensus.cc:2749] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:04.351437 30752 raft_consensus.cc:2468] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:04.330339 30673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 453) took 1085 ms. Trace:
I20251025 20:04:04.305227 31505 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.340894 31461 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717)
I20251025 20:04:04.321810 31532 consensus_queue.cc:1048] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
W20251025 20:04:04.372114 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 46595.7us
I20251025 20:04:04.283013 30671 rpcz_store.cc:276] 1025 20:04:03.230979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.130712 (+899733us) service_pool.cc:225] Handling call
1025 20:04:04.174301 (+ 43589us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.217610 (+ 43309us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.226152 (+ 8542us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.244107 (+ 17955us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.250321 (+ 6214us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.256335 (+ 6014us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.272817 (+ 16482us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1240,"raft.queue_time_us":8575,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":2562}
W20251025 20:04:04.374565 30645 leader_election.cc:343] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:04:04.366075 31460 raft_consensus.cc:2749] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:04:04.352896 31418 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.390198 30645 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:04.340041 31552 consensus_queue.cc:237] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.376921 31318 raft_consensus.cc:3060] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:04.392305 31435 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.365623 31548 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.319079 31559 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.321270 31570 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.372864 31530 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:04.398082 30645 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:04.398902 30800 tablet_bootstrap.cc:700] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: Time spent opening tablet: real 0.127s user 0.034s sys 0.050s
I20251025 20:04:04.399011 30570 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:04.408447 30570 leader_election.cc:304] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:04.380631 31518 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.410706 30800 tablet_bootstrap.cc:654] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:04.375138 31327 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.373383 31491 consensus_queue.cc:1048] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20251025 20:04:04.394896 30672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 292) took 3983 ms. Trace:
I20251025 20:04:04.414186 30570 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:04.396186 30600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 473) took 1032 ms. Trace:
I20251025 20:04:04.425060 31555 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.341545 31549 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.381536 31492 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.381101 30673 rpcz_store.cc:276] 1025 20:04:03.235415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.030711 (+795296us) service_pool.cc:225] Handling call
1025 20:04:04.321062 (+290351us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.402938 31542 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.399235 30604 raft_consensus.cc:2468] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
W20251025 20:04:04.459219 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:04.451252 30796 tablet_bootstrap.cc:654] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:04.468216 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.422403 31481 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.471521 31559 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.472390 31570 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.407375 31530 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:04.412985 31519 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:04.511924 30753 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
W20251025 20:04:04.497206 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 59452.1us
I20251025 20:04:04.474743 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.521971 30750 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:04.421660 31435 raft_consensus.cc:2804] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:04:04.413697 31548 raft_consensus.cc:2804] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:04.435133 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.435894 30677 raft_consensus.cc:1240] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20251025 20:04:04.435197 31568 log.cc:927] Time spent T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20251025 20:04:04.517808 31556 leader_election.cc:290] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.519125 31563 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.467000 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:04.382828 31562 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.436561 30600 rpcz_store.cc:276] 1025 20:04:03.357609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.366201 (+ 8592us) service_pool.cc:225] Handling call
1025 20:04:04.338776 (+972575us) spinlock_profiling.cc:243] Waited 6.19 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea507ce 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:04.389965 (+ 51189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14231424}
I20251025 20:04:04.438354 31493 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.436978 31508 consensus_queue.cc:1048] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.083s
I20251025 20:04:04.366914 31547 consensus_queue.cc:237] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.529996 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.557127 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.505995 31234 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:04.587319 30719 leader_election.cc:343] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:04:04.595854 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:04.434269 30747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 409) took 1756 ms. Trace:
I20251025 20:04:04.468138 31327 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:04.520105 31530 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:04.579401 31572 log.cc:927] Time spent T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.154s user 0.001s sys 0.000s
I20251025 20:04:04.524876 31435 raft_consensus.cc:493] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:04.520480 31519 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:04.527654 31548 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:04.529815 31461 raft_consensus.cc:2749] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:04.434999 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.624065 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.616374 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.618425 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.619442 31455 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.620081 30597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 476) took 1221 ms. Trace:
I20251025 20:04:04.577167 31423 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.577533 30671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 446) took 1344 ms. Trace:
I20251025 20:04:04.430197 31554 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.587388 31391 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.605768 31327 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:04.609114 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:04.595958 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 437) took 1368 ms. Trace:
I20251025 20:04:04.535351 31557 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.612068 31435 raft_consensus.cc:3060] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:04.627267 30597 rpcz_store.cc:276] 1025 20:04:03.398419 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.556418 (+157999us) service_pool.cc:225] Handling call
1025 20:04:03.670140 (+113722us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:03.737230 (+ 67090us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:03.737280 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:04.025844 (+288564us) log.cc:844] Serialized 26 byte log entry
1025 20:04:04.147535 (+121691us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.147579 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.147666 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:04.619846 (+472180us) raft_consensus.cc:1685] finished
1025 20:04:04.619874 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.620017 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19818,"raft.queue_time_us":20282,"spinlock_wait_cycles":8507776,"thread_start_us":59135,"threads_started":1,"wal-append.queue_time_us":102172}
I20251025 20:04:04.622509 31519 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:04.666214 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.621433 31548 raft_consensus.cc:3060] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:04.667972 30570 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34886 dropped due to backpressure: service queue is full with 50 items
I20251025 20:04:04.659543 31505 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.433312 30672 rpcz_store.cc:276] 1025 20:04:00.407407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:00.727705 (+320298us) service_pool.cc:225] Handling call
1025 20:04:00.728327 (+ 622us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:02.021790 (+1293463us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:02.031005 (+ 9215us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:02.162203 (+131198us) log.cc:844] Serialized 26 byte log entry
1025 20:04:02.736407 (+574204us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:02.787891 (+ 51484us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:02.788050 (+ 159us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:04.380595 (+1592545us) raft_consensus.cc:1685] finished
1025 20:04:04.380631 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.390219 (+ 9588us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3466,"raft.queue_time_us":16183,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":32466,"thread_start_us":1522112,"threads_started":2,"wal-append.queue_time_us":1732199}
I20251025 20:04:04.670619 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.670810 30673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 448) took 1437 ms. Trace:
I20251025 20:04:04.642220 30746 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:04:04.678031 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:04.528275 31281 consensus_queue.cc:237] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.635278 30671 rpcz_store.cc:276] 1025 20:04:03.233291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.399954 (+1166663us) service_pool.cc:225] Handling call
1025 20:04:04.496711 (+ 96757us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.521729 (+ 25018us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.526153 (+ 4424us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.528867 (+ 2714us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.538734 (+ 9867us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.556436 (+ 17702us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.575341 (+ 18905us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.664774 30677 rpcz_store.cc:276] 1025 20:04:03.227725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.230032 (+ 2307us) service_pool.cc:225] Handling call
1025 20:04:04.435322 (+1205290us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:04.435585 (+ 263us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:04.583136 (+147551us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.590119 (+ 6983us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9390464}
I20251025 20:04:04.660133 31327 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:04.601151 30747 rpcz_store.cc:276] 1025 20:04:02.665668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.322308 (+1656640us) service_pool.cc:225] Handling call
1025 20:04:04.372179 (+ 49871us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.385314 (+ 13135us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.389840 (+ 4526us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.397810 (+ 7970us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.399555 (+ 1745us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.405548 (+ 5993us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.422224 (+ 16676us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.672152 30599 raft_consensus.cc:2468] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:04.686184 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:04.672760 31428 consensus_meta.cc:287] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 1.672s user 0.016s sys 0.002s
I20251025 20:04:04.701460 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.704969 30753 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:04.701571 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.671316 30673 rpcz_store.cc:276] 1025 20:04:03.233861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.538355 (+1304494us) service_pool.cc:225] Handling call
1025 20:04:04.604584 (+ 66229us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.621308 (+ 16724us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.624207 (+ 2899us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.635449 (+ 11242us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.649402 (+ 13953us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.667751 (+ 18349us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.670735 (+ 2984us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.695281 31378 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.713500 30719 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:04.729224 30678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 439) took 1500 ms. Trace:
I20251025 20:04:04.729319 31234 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:04.729741 30678 rpcz_store.cc:276] 1025 20:04:03.228582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.228725 (+ 143us) service_pool.cc:225] Handling call
1025 20:04:03.229122 (+ 397us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:03.229507 (+ 385us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:03.229532 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:04.353655 (+1124123us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b2c00010828. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8204399b1 00007fd820448fe2 00007fd821b75701 00007fd821b39fda 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22
1025 20:04:04.356533 (+ 2878us) log.cc:844] Serialized 26 byte log entry
1025 20:04:04.528024 (+171491us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.528080 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.530235 (+ 2155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:04.727395 (+197160us) raft_consensus.cc:1685] finished
1025 20:04:04.727422 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.729149 (+ 1727us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":63791,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":3014,"spinlock_wait_cycles":30200192,"thread_start_us":170203,"threads_started":1,"wal-append.queue_time_us":349943}
I20251025 20:04:04.712445 30672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 449) took 1478 ms. Trace:
I20251025 20:04:04.732275 30678 raft_consensus.cc:1275] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:04.733067 30678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 455) took 1497 ms. Trace:
I20251025 20:04:04.729921 30751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 401) took 4733 ms. Trace:
I20251025 20:04:04.733496 30678 rpcz_store.cc:276] 1025 20:04:03.236097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.731501 (+1495404us) service_pool.cc:225] Handling call
1025 20:04:04.731830 (+ 329us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.732739 (+ 909us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.732984 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.716804 31492 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.738309 31428 raft_consensus.cc:2804] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:04.728070 30749 raft_consensus.cc:1275] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:04.738960 31428 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:04.739169 31562 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.739434 31428 raft_consensus.cc:3060] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:04:04.740985 31574 log.cc:927] Time spent T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20251025 20:04:04.750946 30793 tablet_bootstrap.cc:492] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:04:04.750414 31431 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:04.752501 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.734643 31485 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:04.739218 31268 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:04.727703 30670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 443) took 1493 ms. Trace:
I20251025 20:04:04.752025 30679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 445) took 1519 ms. Trace:
I20251025 20:04:04.756610 30793 ts_tablet_manager.cc:1403] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 3.405s user 0.060s sys 0.084s
I20251025 20:04:04.752019 30678 raft_consensus.cc:1275] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:04.734983 31378 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:04.762363 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:04.764182 30678 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:04.668335 30570 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34886 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 489)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 488)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 398)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 487)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 479)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 482)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 492)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 509)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 512)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 513)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 516)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 517)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 518)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 400)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 520)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 402)
I20251025 20:04:04.788805 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.789984 30750 raft_consensus.cc:2368] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:04.792583 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.734545 31234 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.754606 31552 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:04.765663 31378 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.734493 30751 rpcz_store.cc:276] 1025 20:03:59.996733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.534326 (+2537593us) service_pool.cc:225] Handling call
1025 20:04:02.534914 (+ 588us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:03.088233 (+553319us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:03.165960 (+ 77727us) log.cc:844] Serialized 14 byte log entry
1025 20:04:03.496229 (+330269us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:03.537918 (+ 41689us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:03.544425 (+ 6507us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.729607 (+1185182us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b540021fa54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:04.729670 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.729836 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":244149,"prepare.queue_time_us":73779,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":63939,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":22091,"spinlock_wait_cycles":1235521280,"thread_start_us":190307,"threads_started":1,"wal-append.queue_time_us":761016}
I20251025 20:04:04.799089 30674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 442) took 1567 ms. Trace:
I20251025 20:04:04.805338 30674 rpcz_store.cc:276] 1025 20:04:03.231473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.136850 (+905377us) service_pool.cc:225] Handling call
1025 20:04:04.166066 (+ 29216us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:04.176601 (+ 10535us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.176651 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:04.187624 (+ 10973us) log.cc:844] Serialized 26 byte log entry
1025 20:04:04.274598 (+ 86974us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.274629 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.275880 (+ 1251us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:04.798776 (+522896us) raft_consensus.cc:1685] finished
1025 20:04:04.798807 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.798988 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9982,"raft.queue_time_us":10194,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":22752,"thread_start_us":78347,"threads_started":1,"wal-append.queue_time_us":341087}
I20251025 20:04:04.814633 30793 raft_consensus.cc:359] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.815822 30793 raft_consensus.cc:385] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:04.816167 30793 raft_consensus.cc:740] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:04:04.817062 30793 consensus_queue.cc:260] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.758884 31268 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.803433 31552 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:04.824784 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:04.804989 30673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 454) took 1569 ms. Trace:
I20251025 20:04:04.735873 30672 rpcz_store.cc:276] 1025 20:04:03.234159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.671196 (+1437037us) service_pool.cc:225] Handling call
1025 20:04:04.671589 (+ 393us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.678248 (+ 6659us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.679595 (+ 1347us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.687845 (+ 8250us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.696246 (+ 8401us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.701376 (+ 5130us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.709237 (+ 7861us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.824337 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:04.839538 30752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 333) took 1526 ms (client timeout 1877 ms). Trace:
W20251025 20:04:04.839955 30752 rpcz_store.cc:269] 1025 20:04:03.313223 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.016786 (+703563us) service_pool.cc:225] Handling call
1025 20:04:04.839341 (+822555us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b5400287654. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:04.839475 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":505514368}
I20251025 20:04:04.835919 30570 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34886 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 489)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 488)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 398)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 487)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 479)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 482)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 492)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 509)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 512)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 513)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 516)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 517)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 518)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 400)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 520)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 402)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 521)
I20251025 20:04:04.756217 30670 rpcz_store.cc:276] 1025 20:04:03.231943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.282823 (+1050880us) service_pool.cc:225] Handling call
1025 20:04:04.283228 (+ 405us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.283467 (+ 239us) raft_consensus.cc:1534] Early marking committed up to index 1
1025 20:04:04.529363 (+245896us) log.cc:844] Serialized 14 byte log entry
1025 20:04:04.706622 (+177259us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.709079 (+ 2457us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.710487 (+ 1408us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.714826 (+ 4339us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.725147 (+ 10321us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":403617,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":2715,"raft.queue_time_us":4999,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"thread_start_us":410872,"threads_started":2}
W20251025 20:04:04.921794 30602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 466) took 1667 ms (client timeout 1962 ms). Trace:
I20251025 20:04:04.841444 30570 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34886 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 489)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 488)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 398)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 487)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 479)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 482)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 492)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 509)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 512)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 513)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 516)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 517)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 518)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 400)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 520)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 402)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 521)
I20251025 20:04:04.927510 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:04.833475 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 242294us
W20251025 20:04:04.841465 30676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 447) took 1607 ms (client timeout 1727 ms). Trace:
W20251025 20:04:04.931941 30676 rpcz_store.cc:269] 1025 20:04:03.233576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.369875 (+136299us) service_pool.cc:225] Handling call
1025 20:04:04.841318 (+1471443us) spinlock_profiling.cc:243] Waited 61 ms on lock 0x7b54000a78d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:04.841409 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":140391168}
I20251025 20:04:04.933564 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:04.900868 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:04.935874 30645 connection.cc:232] RPC call timeout handler was delayed by -1.0925399460002154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
W20251025 20:04:04.936479 30645 connection.cc:232] RPC call timeout handler was delayed by -1.091639946000214s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
W20251025 20:04:04.936973 30645 connection.cc:232] RPC call timeout handler was delayed by -1.0619399460001659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
W20251025 20:04:04.922322 30602 rpcz_store.cc:269] 1025 20:04:03.174403 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.174567 (+ 164us) service_pool.cc:225] Handling call
1025 20:04:04.841086 (+1666519us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b540012f2d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:04.841188 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":277295360}
I20251025 20:04:04.829119 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:04.995600 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.064013 31552 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.071164 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:04.832981 31549 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:05.006000 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:05.072850 30596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 475) took 1689 ms (client timeout 1760 ms). Trace:
I20251025 20:04:05.073105 30674 raft_consensus.cc:2376] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1. Re-sending same reply.
W20251025 20:04:05.073370 30596 rpcz_store.cc:269] 1025 20:04:03.383374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.405457 (+ 22083us) service_pool.cc:225] Handling call
1025 20:04:04.824758 (+1419301us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:05.072773 (+248015us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4351232}
I20251025 20:04:05.074499 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.074642 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:04.998072 30745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 316) took 4895 ms. Trace:
I20251025 20:04:04.996253 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.076223 30745 rpcz_store.cc:276] 1025 20:04:00.102944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.261689 (+3158745us) service_pool.cc:225] Handling call
1025 20:04:03.351226 (+ 89537us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.073146 (+721920us) raft_consensus.cc:1534] Early marking committed up to index 1
1025 20:04:04.158242 (+ 85096us) log.cc:844] Serialized 14 byte log entry
1025 20:04:04.206082 (+ 47840us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.206147 (+ 65us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.206193 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.997599 (+791406us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b54002d8554. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:04.997689 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.997975 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":131939,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":109,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"spinlock_wait_cycles":1564093696,"thread_start_us":117364,"threads_started":2,"wal-append.queue_time_us":47451}
I20251025 20:04:05.075305 30674 raft_consensus.cc:2468] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:04:05.013489 30570 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34874 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 488)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 398)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 487)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 479)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 482)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 492)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 509)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 512)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 513)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 516)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 517)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 518)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 400)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 520)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 402)
I20251025 20:04:05.084349 30570 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34886 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 488)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 398)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 411)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 412)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 487)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 479)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 482)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 492)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 509)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 512)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 513)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 516)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 517)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 518)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 400)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 520)
I20251025 20:04:05.094365 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.090030 30570 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34874 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 488)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 398)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 411)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 413)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 412)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 487)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 479)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 482)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 492)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 509)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 512)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 513)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 516)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 517)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 518)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 400)
I20251025 20:04:05.095906 30793 ts_tablet_manager.cc:1434] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.338s user 0.013s sys 0.002s
I20251025 20:04:05.078348 30676 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
W20251025 20:04:05.064196 31578 log.cc:927] Time spent T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.297s user 0.001s sys 0.000s
I20251025 20:04:05.054324 31414 raft_consensus.cc:493] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:05.057610 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:04.988243 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.115860 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.117774 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.117993 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:05.118206 30676 raft_consensus.cc:2468] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:05.094687 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.121196 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.121821 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.075305 30596 raft_consensus.cc:2468] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:05.120260 30793 tablet_bootstrap.cc:492] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:04:05.102640 30595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 469) took 1868 ms. Trace:
I20251025 20:04:05.136355 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.137548 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 488) took 1047 ms. Trace:
I20251025 20:04:05.133273 30570 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34886 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 398)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 411)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 416)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 413)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 412)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 487)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 393)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 414)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 479)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 482)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 492)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 509)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 512)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 513)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 516)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 517)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 518)
I20251025 20:04:05.138037 30602 rpcz_store.cc:276] 1025 20:04:04.090009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.125020 (+1035011us) service_pool.cc:225] Handling call
1025 20:04:05.137463 (+ 12443us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:04.828938 30673 rpcz_store.cc:276] 1025 20:04:03.235776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.719195 (+1483419us) service_pool.cc:225] Handling call
1025 20:04:04.723779 (+ 4584us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.756135 (+ 32356us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.759491 (+ 3356us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.760949 (+ 1458us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.767242 (+ 6293us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.786686 (+ 19444us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.803901 (+ 17215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.140563 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:05.131705 30675 consensus_meta.cc:287] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.919s user 0.009s sys 0.006s
I20251025 20:04:05.144686 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.145866 30745 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:05.128782 31234 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.131889 30596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 489) took 1035 ms. Trace:
I20251025 20:04:05.132892 31378 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.121186 31414 raft_consensus.cc:515] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.165618 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.155297 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.167250 30602 raft_consensus.cc:2468] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:05.167955 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 491) took 1056 ms. Trace:
I20251025 20:04:05.168387 30602 rpcz_store.cc:276] 1025 20:04:04.111806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.139106 (+1027300us) service_pool.cc:225] Handling call
1025 20:04:05.167866 (+ 28760us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.125854 30751 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:05.142536 31551 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:05.169981 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.177865 30745 raft_consensus.cc:2393] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:05.147812 30675 raft_consensus.cc:2468] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:05.182407 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:05.177553 31585 log.cc:927] Time spent T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.345s user 0.002s sys 0.000s
I20251025 20:04:05.187889 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.178385 30671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 451) took 1939 ms. Trace:
I20251025 20:04:05.158273 30596 rpcz_store.cc:276] 1025 20:04:04.096744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.074392 (+977648us) service_pool.cc:225] Handling call
1025 20:04:05.127780 (+ 53388us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:05.189142 31584 log.cc:927] Time spent T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.268s user 0.001s sys 0.000s
I20251025 20:04:05.199079 30796 tablet_bootstrap.cc:492] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:04:05.181732 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.201390 30796 ts_tablet_manager.cc:1403] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.894s user 0.027s sys 0.050s
I20251025 20:04:05.201202 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:05.190347 30675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 440) took 1955 ms (client timeout 1962 ms). Trace:
I20251025 20:04:05.205138 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.206801 30800 tablet_bootstrap.cc:492] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:04:05.206662 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 496) took 1070 ms. Trace:
I20251025 20:04:05.179929 31551 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.207293 30800 ts_tablet_manager.cc:1403] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 3.189s user 0.060s sys 0.075s
I20251025 20:04:05.207135 30602 rpcz_store.cc:276] 1025 20:04:04.136121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.198032 (+1061911us) service_pool.cc:225] Handling call
1025 20:04:05.206578 (+ 8546us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.200944 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.213982 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.215407 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.194856 31428 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.208024 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.221014 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.206296 30673 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:05.213482 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.219543 30670 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:05.228375 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.237690 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.267035 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.265551 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:05.207602 30675 rpcz_store.cc:269] 1025 20:04:03.230509 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.751660 (+521151us) service_pool.cc:225] Handling call
1025 20:04:05.186150 (+1434490us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.267346 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.323144 30753 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:05.327695 30796 raft_consensus.cc:359] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.329291 30796 raft_consensus.cc:385] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:05.329855 30796 raft_consensus.cc:740] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
W20251025 20:04:05.297216 31590 log.cc:927] Time spent T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
W20251025 20:04:05.233454 30744 consensus_meta.cc:287] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 1.159s user 0.010s sys 0.004s
I20251025 20:04:05.331146 30796 consensus_queue.cc:260] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:05.317324 31591 log.cc:927] Time spent T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
W20251025 20:04:05.231295 31318 consensus_meta.cc:287] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.832s user 0.013s sys 0.003s
I20251025 20:04:05.334697 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:05.330271 31592 log.cc:927] Time spent T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20251025 20:04:05.240296 31268 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:05.249732 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.266144 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.266871 31428 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:05.234368 31414 leader_election.cc:290] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:05.317999 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.338807 31318 raft_consensus.cc:2749] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:04:05.343137 30678 raft_consensus.cc:2468] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:05.343711 30796 ts_tablet_manager.cc:1434] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.141s user 0.009s sys 0.001s
W20251025 20:04:05.348462 31548 consensus_meta.cc:287] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.669s user 0.008s sys 0.008s
I20251025 20:04:05.348544 30793 tablet_bootstrap.cc:654] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
W20251025 20:04:05.348749 31594 log.cc:927] Time spent T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20251025 20:04:05.349948 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.335813 30744 raft_consensus.cc:2468] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:04:05.351159 30676 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:05.343540 31551 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.351212 30670 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:05.352336 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:05.360361 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 472) took 2041 ms. Trace:
W20251025 20:04:05.372205 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 67241.4us
W20251025 20:04:05.359207 31327 consensus_meta.cc:287] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.665s user 0.018s sys 0.003s
W20251025 20:04:05.360983 31519 consensus_meta.cc:287] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.681s user 0.010s sys 0.002s
W20251025 20:04:05.361153 30744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 411) took 2690 ms (client timeout 3183 ms). Trace:
I20251025 20:04:05.363301 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:05.341190 31595 log.cc:927] Time spent T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20251025 20:04:05.197839 30671 rpcz_store.cc:276] 1025 20:04:03.234738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.704805 (+1470067us) service_pool.cc:225] Handling call
1025 20:04:04.735222 (+ 30417us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.759582 (+ 24360us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:04.807325 (+ 47743us) log.cc:844] Serialized 14 byte log entry
1025 20:04:05.143129 (+335804us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:05.148248 (+ 5119us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:05.152175 (+ 3927us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:05.156363 (+ 4188us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:05.174096 (+ 17733us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":58252,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":1758,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":7299,"thread_start_us":309121,"threads_started":1,"wal-append.queue_time_us":358787}
I20251025 20:04:05.398504 31519 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.344483 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.403070 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.348945 31548 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.399883 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.422021 30643 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41100 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 322)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 326)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 316)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 325)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 319)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 299)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 327)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 328)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 323)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 334)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 338)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 335)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 333)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 324)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 339)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 321)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 337)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 329)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 477)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 485)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 314)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 332)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 313)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 462)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 463)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 469)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 474)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 488)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 491)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 493)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 312)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 320)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 330)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 336)
I20251025 20:04:05.370440 30603 rpcz_store.cc:276] 1025 20:04:03.318720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.380245 (+ 61525us) service_pool.cc:225] Handling call
1025 20:04:04.420823 (+1040578us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:04.421054 (+ 231us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.421277 (+ 223us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:04.435412 (+ 14135us) log.cc:844] Serialized 14 byte log entry
1025 20:04:05.359645 (+924233us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b18000ba160. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8278d09a1 00007fd8278dd371 00007fd8216e255c 00007fd81ea7097b 00007fd827849219 00007fd81ea8cc4b 00007fd81ea8aeef 00007fd81ea8d1b2 00007fd8216d42bb 00007fd8216d9dc3 00007fd8216da2da
1025 20:04:05.359985 (+ 340us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:05.360046 (+ 61us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:05.360092 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:05.360175 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:05.360298 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16710,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":112,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":120,"spinlock_wait_cycles":65328128,"thread_start_us":66129,"threads_started":1,"wal-append.queue_time_us":130316}
I20251025 20:04:05.426695 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:05.427312 30603 raft_consensus.cc:2376] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4. Re-sending same reply.
I20251025 20:04:05.427785 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 504) took 1236 ms. Trace:
I20251025 20:04:05.428102 30603 rpcz_store.cc:276] 1025 20:04:04.191730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.426403 (+1234673us) service_pool.cc:225] Handling call
1025 20:04:05.427732 (+ 1329us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.428828 31499 consensus_queue.cc:1059] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:05.425107 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.431613 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.431730 31548 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.432600 30675 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:05.434054 31519 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.433992 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:05.397274 30744 rpcz_store.cc:269] 1025 20:04:02.670661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.200114 (+529453us) service_pool.cc:225] Handling call
1025 20:04:05.361042 (+2160928us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.429802 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:05.435952 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:05.355921 31435 consensus_meta.cc:287] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.683s user 0.007s sys 0.005s
I20251025 20:04:05.436671 30748 raft_consensus.cc:2376] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2. Re-sending same reply.
I20251025 20:04:05.428954 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.429123 30800 raft_consensus.cc:359] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.396324 31327 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.438131 30800 raft_consensus.cc:385] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:05.438244 30671 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:05.438596 30800 raft_consensus.cc:740] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:04:05.439306 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.440230 31327 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.440691 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.437306 31435 raft_consensus.cc:515] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.444160 31435 leader_election.cc:290] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:05.434129 31530 consensus_meta.cc:287] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.808s user 0.010s sys 0.009s
I20251025 20:04:05.446035 30645 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:05.447454 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 79747.5us
I20251025 20:04:05.446200 31530 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.449085 31519 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:05.448935 31499 consensus_queue.cc:1048] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:05.449581 31519 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:05.450049 31519 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:05.451743 30716 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.452929 31414 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:05.453327 31414 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:05.453678 31414 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:05.453559 31530 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.442206 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.437669 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.456384 30671 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:05.457474 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.448427 30645 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.437810 30603 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:05.457939 31268 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:05.458360 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 495) took 1324 ms. Trace:
I20251025 20:04:05.458463 31268 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.458583 30671 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:05.459326 30716 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:05.458741 30603 rpcz_store.cc:276] 1025 20:04:04.133930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.428706 (+1294776us) service_pool.cc:225] Handling call
1025 20:04:05.458296 (+ 29590us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.461994 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.462779 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 498) took 1314 ms. Trace:
I20251025 20:04:05.463102 30603 rpcz_store.cc:276] 1025 20:04:04.148323 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.461674 (+1313351us) service_pool.cc:225] Handling call
1025 20:04:05.462720 (+ 1046us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.463450 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.464049 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.464816 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.465207 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.439709 30800 consensus_queue.cc:260] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.460573 31268 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:05.466105 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.466827 30675 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:05.467535 30800 ts_tablet_manager.cc:1434] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.260s user 0.006s sys 0.005s
I20251025 20:04:05.467809 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.468675 30800 tablet_bootstrap.cc:492] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:04:05.472311 31318 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:05.472826 31318 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:05.473176 31318 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:05.473493 31499 raft_consensus.cc:2749] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:05.474875 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 499) took 1287 ms. Trace:
I20251025 20:04:05.475275 30603 rpcz_store.cc:276] 1025 20:04:04.187100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.464952 (+1277852us) service_pool.cc:225] Handling call
1025 20:04:05.474796 (+ 9844us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.476234 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.477299 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.460290 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.477873 30603 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:05.482896 30645 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.480868 30716 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:05.488144 30716 leader_election.cc:304] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.486304 30645 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.469084 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.495620 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 452) took 2260 ms. Trace:
I20251025 20:04:05.494279 30645 leader_election.cc:400] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:05.498270 30645 leader_election.cc:403] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:04:05.490875 31481 raft_consensus.cc:2804] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:05.497188 31290 raft_consensus.cc:2804] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:05.498176 30675 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:05.491914 31485 raft_consensus.cc:2804] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:04:05.496091 30677 rpcz_store.cc:276] 1025 20:04:03.235072 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.493172 (+2258100us) service_pool.cc:225] Handling call
1025 20:04:05.494975 (+ 1803us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:05.495190 (+ 215us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:05.495221 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:05.495274 (+ 53us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:05.495313 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:05.495377 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:05.495512 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.499361 31268 raft_consensus.cc:2804] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:04:05.505771 31481 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:05.507824 30716 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.507313 31290 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:05.532889 31290 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:05.533530 31519 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.534593 30793 tablet_bootstrap.cc:492] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:04:05.535192 30793 ts_tablet_manager.cc:1403] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 0.428s user 0.022s sys 0.013s
I20251025 20:04:05.535606 31485 raft_consensus.cc:697] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:05.485437 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 510) took 1101 ms. Trace:
I20251025 20:04:05.537423 30603 rpcz_store.cc:276] 1025 20:04:04.384330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.477065 (+1092735us) service_pool.cc:225] Handling call
1025 20:04:05.485363 (+ 8298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.539719 30645 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.541209 31563 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:05.541637 31563 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:05.543401 31563 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:05.546691 31318 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.549098 31318 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.550303 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:05.552428 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.508132 31552 raft_consensus.cc:2749] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:05.550468 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.542929 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.555444 30645 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.556504 30800 tablet_bootstrap.cc:654] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:05.553256 30677 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:05.598069 30603 raft_consensus.cc:2393] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:05.506453 30671 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:05.615904 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 399) took 1011 ms. Trace:
I20251025 20:04:05.525957 31268 raft_consensus.cc:493] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:05.616582 30603 rpcz_store.cc:276] 1025 20:04:04.605002 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.542540 (+937538us) service_pool.cc:225] Handling call
1025 20:04:05.615807 (+ 73267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.536293 31481 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:05.627370 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.631888 30603 raft_consensus.cc:2468] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:05.633006 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 508) took 1262 ms. Trace:
I20251025 20:04:05.633446 30603 rpcz_store.cc:276] 1025 20:04:04.371066 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.619847 (+1248781us) service_pool.cc:225] Handling call
1025 20:04:05.632934 (+ 13087us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.621282 31414 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.627053 31290 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.617697 31318 raft_consensus.cc:2804] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:05.641619 30645 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.617254 30749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 322) took 5226 ms. Trace:
I20251025 20:04:05.619530 31549 raft_consensus.cc:2804] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:05.621551 31268 raft_consensus.cc:3060] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:04:05.645723 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 108425us
I20251025 20:04:05.628762 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.618539 30675 raft_consensus.cc:2468] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:05.639717 31414 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:05.649559 30716 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:05.626806 31519 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.643589 31318 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:05.650058 31549 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:05.772177 31318 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:05.673956 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.785681 31531 raft_consensus.cc:2804] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:05.629163 31485 consensus_queue.cc:237] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.788698 31549 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:05.834851 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.809754 30677 raft_consensus.cc:2393] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 4: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:05.796283 31604 raft_consensus.cc:2804] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:05.835583 30603 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:04:05.836391 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 407) took 1002 ms. Trace:
I20251025 20:04:05.836747 30603 rpcz_store.cc:276] 1025 20:04:04.833420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.642211 (+808791us) service_pool.cc:225] Handling call
1025 20:04:05.836049 (+193838us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.646816 30749 rpcz_store.cc:276] 1025 20:04:00.390607 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.180976 (+3790369us) service_pool.cc:225] Handling call
1025 20:04:04.218784 (+ 37808us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.738994 (+520210us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:05.536377 (+797383us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b540020ab54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:05.558953 (+ 22576us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1129026432}
I20251025 20:04:05.841223 30793 raft_consensus.cc:359] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.808750 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.841933 30793 raft_consensus.cc:385] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:05.844889 30793 raft_consensus.cc:740] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:04:05.811297 31531 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:05.845489 30793 consensus_queue.cc:260] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.842561 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.852234 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.837332 31604 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
W20251025 20:04:05.853741 31603 log.cc:927] Time spent T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.178s user 0.000s sys 0.001s
I20251025 20:04:05.855996 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 471) took 2598 ms. Trace:
I20251025 20:04:05.855863 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.857941 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 497) took 1714 ms. Trace:
I20251025 20:04:05.861649 30603 rpcz_store.cc:276] 1025 20:04:04.143751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.851962 (+1708211us) service_pool.cc:225] Handling call
1025 20:04:05.857834 (+ 5872us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.845254 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 470) took 1024 ms. Trace:
I20251025 20:04:05.844213 31290 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.850265 31531 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:05.858441 30793 ts_tablet_manager.cc:1434] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.323s user 0.007s sys 0.004s
I20251025 20:04:05.863085 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.850699 30675 raft_consensus.cc:2468] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:04:05.859158 31604 raft_consensus.cc:3060] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:05.869664 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.864399 30671 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:05.873134 30603 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 6: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:05.873660 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 404) took 1237 ms. Trace:
I20251025 20:04:05.874145 30603 rpcz_store.cc:276] 1025 20:04:04.636247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.869288 (+1233041us) service_pool.cc:225] Handling call
1025 20:04:05.873573 (+ 4285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.875380 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.876798 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.877559 30603 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:05.878533 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.879297 30603 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:05.863323 30677 rpcz_store.cc:276] 1025 20:04:04.821842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.628415 (+806573us) service_pool.cc:225] Handling call
1025 20:04:05.842502 (+214087us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.884783 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.868983 31563 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.860579 30598 rpcz_store.cc:276] 1025 20:04:03.257450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.260194 (+ 2744us) service_pool.cc:225] Handling call
1025 20:04:03.405043 (+144849us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.077158 (+672115us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.078928 (+ 1770us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.083819 (+ 4891us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.087446 (+ 3627us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:05.855670 (+1768224us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b54001225d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:05.855729 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:05.855900 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1429205760}
I20251025 20:04:05.890206 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:05.891141 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 53355.4us
I20251025 20:04:05.889003 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.895999 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.896893 30603 raft_consensus.cc:2468] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:05.898149 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.898829 30677 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:05.899713 31563 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.899896 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.900566 30677 raft_consensus.cc:2468] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:05.901523 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.903419 31318 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.903887 31549 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.905623 31318 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.906304 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.906764 31549 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:05.907025 30598 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:05.907224 30645 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.907708 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 500) took 1719 ms. Trace:
I20251025 20:04:05.908224 30598 rpcz_store.cc:276] 1025 20:04:04.188658 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.905963 (+1717305us) service_pool.cc:225] Handling call
1025 20:04:05.907628 (+ 1665us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.890359 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.910967 30671 raft_consensus.cc:2468] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:04:05.912223 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.909185 31318 raft_consensus.cc:2804] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:05.909523 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.922394 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.923334 30603 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:05.913767 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.925865 31318 raft_consensus.cc:697] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:05.921268 30675 raft_consensus.cc:2468] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:04:05.937222 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.938694 30603 raft_consensus.cc:2368] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 5. Current term is 6.
I20251025 20:04:05.939816 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 487) took 1855 ms. Trace:
I20251025 20:04:05.940179 30603 rpcz_store.cc:276] 1025 20:04:04.084121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.926986 (+1842865us) service_pool.cc:225] Handling call
1025 20:04:05.939749 (+ 12763us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:05.940202 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 171834us
W20251025 20:04:05.942281 30719 leader_election.cc:336] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.739s (SENT)
I20251025 20:04:05.931298 30671 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:05.927959 30598 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:05.946180 31541 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.946346 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:05.947041 30749 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:05.947119 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:05.947775 30746 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 6
I20251025 20:04:05.941802 31423 raft_consensus.cc:3055] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:05.943756 31531 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.944173 30716 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.951107 31423 raft_consensus.cc:740] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:05.960287 31531 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.960223 31423 consensus_queue.cc:260] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.961097 31209 raft_consensus.cc:3055] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:05.961490 31209 raft_consensus.cc:740] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:05.961664 31423 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:05.944358 31268 raft_consensus.cc:515] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.965386 30746 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 6.
I20251025 20:04:05.965579 31614 raft_consensus.cc:2804] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:05.966275 31614 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:05.966387 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.966667 31614 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:05.967715 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.968591 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.954347 30716 leader_election.cc:304] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:05.969302 30675 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:05.970278 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.970377 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:05.967803 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.971169 30748 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:05.971320 30675 raft_consensus.cc:2393] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 5: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:05.971320 30674 raft_consensus.cc:2468] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:04:05.971894 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:05.972432 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.972697 30744 raft_consensus.cc:2393] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:05.972787 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:05.973213 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:05.973425 30752 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:05.973665 30674 raft_consensus.cc:2393] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 4: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:05.970237 30716 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:05.975060 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.976181 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.976930 30674 raft_consensus.cc:2393] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:05.977699 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.978060 31541 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:05.978144 30716 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.978758 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.978662 31541 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.979367 30674 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:05.979468 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 514) took 1506 ms. Trace:
I20251025 20:04:05.979791 30603 rpcz_store.cc:276] 1025 20:04:04.473578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.943087 (+1469509us) service_pool.cc:225] Handling call
1025 20:04:05.979404 (+ 36317us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.980273 30716 leader_election.cc:304] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.980644 31541 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:05.980700 30674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 462) took 1174 ms. Trace:
I20251025 20:04:05.980677 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.981374 31414 raft_consensus.cc:2804] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:04:05.981494 30603 raft_consensus.cc:2468] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:05.981747 30716 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:05.981979 31414 raft_consensus.cc:493] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:05.982144 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 507) took 1611 ms. Trace:
I20251025 20:04:05.982435 31414 raft_consensus.cc:3060] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:05.983511 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.984261 30598 raft_consensus.cc:2368] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:05.985069 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:05.986120 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.986310 31607 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:05.986737 31607 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:05.987095 31607 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:05.988008 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 463) took 1180 ms. Trace:
I20251025 20:04:05.988402 30677 rpcz_store.cc:276] 1025 20:04:04.807237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.987097 (+1179860us) service_pool.cc:225] Handling call
1025 20:04:05.987456 (+ 359us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:05.987652 (+ 196us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:05.987680 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:05.987735 (+ 55us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:05.987770 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:05.987830 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:05.987949 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.990468 31268 leader_election.cc:290] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:05.990236 31209 consensus_queue.cc:260] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.991479 31209 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:05.972453 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:05.983000 30716 leader_election.cc:304] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:05.995533 31268 raft_consensus.cc:2804] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:04:05.995955 31268 raft_consensus.cc:493] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:05.996309 31268 raft_consensus.cc:3060] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:05.981238 30674 rpcz_store.cc:276] 1025 20:04:04.806273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.979958 (+1173685us) service_pool.cc:225] Handling call
1025 20:04:05.980226 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:05.980392 (+ 166us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:05.980428 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:05.980473 (+ 45us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:05.980507 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:05.980557 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:05.980653 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:05.982549 30603 rpcz_store.cc:276] 1025 20:04:04.370669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.980434 (+1609765us) service_pool.cc:225] Handling call
1025 20:04:05.982085 (+ 1651us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.004774 31607 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.007519 31607 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:05.125837 30595 rpcz_store.cc:276] 1025 20:04:03.230395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.232800 (+ 2405us) service_pool.cc:225] Handling call
1025 20:04:04.698897 (+1466097us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:04.706228 (+ 7331us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.709038 (+ 2810us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:04.719244 (+ 10206us) log.cc:844] Serialized 14 byte log entry
1025 20:04:05.013301 (+294057us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:05.056445 (+ 43144us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:05.057985 (+ 1540us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:05.065749 (+ 7764us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:05.098650 (+ 32901us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":8591,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":9169,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":153,"spinlock_wait_cycles":3174272,"thread_start_us":261843,"threads_started":1,"wal-append.queue_time_us":420449}
I20251025 20:04:06.009461 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:06.013002 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:06.014274 30595 raft_consensus.cc:3060] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 2
I20251025 20:04:06.014214 31318 consensus_queue.cc:237] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.016664 31268 raft_consensus.cc:515] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.017935 31481 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.019634 30596 raft_consensus.cc:2468] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:06.019965 31604 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:05.999073 31549 raft_consensus.cc:2749] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:06.020656 30671 raft_consensus.cc:2393] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:06.020288 31481 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:06.020287 30596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 515) took 1499 ms (client timeout 1619 ms). Trace:
I20251025 20:04:06.022253 30602 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
W20251025 20:04:06.022857 30602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 398) took 1419 ms (client timeout 1550 ms). Trace:
W20251025 20:04:06.023222 30602 rpcz_store.cc:269] 1025 20:04:04.603252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.211957 (+608705us) service_pool.cc:225] Handling call
1025 20:04:06.022790 (+810833us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.023967 30800 tablet_bootstrap.cc:492] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:04:06.024222 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.024514 30800 ts_tablet_manager.cc:1403] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.556s user 0.022s sys 0.014s
I20251025 20:04:06.024906 30602 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
W20251025 20:04:06.022064 30596 rpcz_store.cc:269] 1025 20:04:04.520765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.204019 (+683254us) service_pool.cc:225] Handling call
1025 20:04:06.020209 (+816190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.027415 30678 raft_consensus.cc:1240] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20251025 20:04:06.028225 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 45297.7us
I20251025 20:04:06.028767 30678 raft_consensus.cc:1240] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
W20251025 20:04:06.031515 30716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:05.985682 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 395) took 1402 ms. Trace:
I20251025 20:04:06.010046 30746 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 3
I20251025 20:04:06.036213 30752 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:06.035144 30598 rpcz_store.cc:276] 1025 20:04:04.583701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.984817 (+1401116us) service_pool.cc:225] Handling call
1025 20:04:05.985631 (+ 814us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.040622 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 480) took 2107 ms. Trace:
I20251025 20:04:06.040418 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:06.041051 30598 rpcz_store.cc:276] 1025 20:04:03.933374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.039699 (+2106325us) service_pool.cc:225] Handling call
1025 20:04:06.040064 (+ 365us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.040252 (+ 188us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.040285 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.040329 (+ 44us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.040368 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.040426 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.040563 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.063544 31604 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:06.064123 30749 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:06.066504 30800 raft_consensus.cc:359] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.067335 30800 raft_consensus.cc:385] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:06.067608 30800 raft_consensus.cc:740] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:04:06.041328 30751 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:06.041726 31610 raft_consensus.cc:2764] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251025 20:04:06.070590 30719 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.071763 30719 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:06.075114 31268 leader_election.cc:290] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:06.075311 30595 raft_consensus.cc:2468] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:04:06.076030 30595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 393) took 1500 ms. Trace:
I20251025 20:04:06.076099 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.076529 30595 rpcz_store.cc:276] 1025 20:04:04.576000 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.012654 (+1436654us) service_pool.cc:225] Handling call
1025 20:04:06.075942 (+ 63288us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.077067 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 401) took 1453 ms. Trace:
I20251025 20:04:06.077437 30603 rpcz_store.cc:276] 1025 20:04:04.623352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.003766 (+1380414us) service_pool.cc:225] Handling call
1025 20:04:06.076998 (+ 73232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.059924 30753 raft_consensus.cc:1275] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:04:06.078907 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 484) took 2004 ms. Trace:
I20251025 20:04:06.069120 30800 consensus_queue.cc:260] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.079636 31532 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:06.080296 31532 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.081487 30800 ts_tablet_manager.cc:1434] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.057s user 0.008s sys 0.000s
I20251025 20:04:06.027437 30596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 479) took 2182 ms. Trace:
I20251025 20:04:06.060482 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:06.033015 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 469) took 1223 ms. Trace:
I20251025 20:04:04.756290 30679 rpcz_store.cc:276] 1025 20:04:03.232536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.328744 (+1096208us) service_pool.cc:225] Handling call
1025 20:04:04.346011 (+ 17267us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.346302 (+ 291us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:04.540781 (+194479us) log.cc:844] Serialized 14 byte log entry
1025 20:04:04.751633 (+210852us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.751694 (+ 61us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:04.751739 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:04.751814 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:04.751954 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83,"prepare.queue_time_us":367859,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":3149,"thread_start_us":388564,"threads_started":2}
I20251025 20:04:06.079243 30603 rpcz_store.cc:276] 1025 20:04:04.075060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.078100 (+2003040us) service_pool.cc:225] Handling call
1025 20:04:06.078399 (+ 299us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.078571 (+ 172us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.078600 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.078647 (+ 47us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.078679 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.078746 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.078848 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.091837 30719 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:06.044373 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.094642 30678 raft_consensus.cc:2393] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
W20251025 20:04:06.099118 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 55104.5us
I20251025 20:04:06.099776 30719 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:06.097826 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.099920 30645 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.082845 30596 rpcz_store.cc:276] 1025 20:04:03.845274 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.026515 (+2181241us) service_pool.cc:225] Handling call
1025 20:04:06.026876 (+ 361us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.027066 (+ 190us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.027101 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.027162 (+ 61us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:06.027197 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.027262 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.027381 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.084245 30677 rpcz_store.cc:276] 1025 20:04:04.809597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.989549 (+1179952us) service_pool.cc:225] Handling call
1025 20:04:05.989885 (+ 336us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.032600 (+ 42715us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.032638 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.032693 (+ 55us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:06.032739 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.032809 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.032949 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.101953 30678 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 6: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:06.105090 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.105839 30679 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:06.106724 30800 tablet_bootstrap.cc:492] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:04:06.105981 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.104952 30645 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.113085 30716 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.097290 31606 raft_consensus.cc:2804] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:06.116024 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 485) took 2037 ms. Trace:
I20251025 20:04:06.064163 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 482) took 2043 ms. Trace:
I20251025 20:04:06.134215 30603 rpcz_store.cc:276] 1025 20:04:04.078488 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.090877 (+2012389us) service_pool.cc:225] Handling call
1025 20:04:06.115471 (+ 24594us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.115677 (+ 206us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.115709 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.115754 (+ 45us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.115784 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.115835 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.115949 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.122139 30678 raft_consensus.cc:2393] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:06.109198 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.097036 31607 raft_consensus.cc:2749] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:06.138767 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:06.100358 30748 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 5.
I20251025 20:04:06.109493 31541 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader election won for term 6
I20251025 20:04:06.127375 30679 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:06.155225 30678 raft_consensus.cc:2468] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:06.112327 31209 raft_consensus.cc:2804] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:06.060741 31414 raft_consensus.cc:515] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.103720 31192 consensus_queue.cc:1048] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
I20251025 20:04:06.165594 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:06.163707 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.166860 30678 raft_consensus.cc:3060] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 3
I20251025 20:04:06.137002 31563 raft_consensus.cc:2804] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:06.096784 31614 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.112196 30746 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:06.163334 30677 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:06.137424 31604 raft_consensus.cc:2804] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:06.166569 30679 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:06.154625 30595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 483) took 2114 ms. Trace:
I20251025 20:04:06.173349 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.174782 30679 raft_consensus.cc:3060] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:06.138916 30598 rpcz_store.cc:276] 1025 20:04:04.020653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.063167 (+2042514us) service_pool.cc:225] Handling call
1025 20:04:06.063512 (+ 345us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.063724 (+ 212us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.063765 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.063820 (+ 55us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.063869 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.063943 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.064092 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:06.178561 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 64703.4us
I20251025 20:04:06.163761 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:06.183523 30675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 491) took 1031 ms. Trace:
I20251025 20:04:06.163995 31209 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:06.166603 31423 raft_consensus.cc:2804] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:06.186017 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.186956 30598 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:06.188967 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.190770 30678 raft_consensus.cc:2468] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:06.192705 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.193531 30678 raft_consensus.cc:3060] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:06.197947 30679 raft_consensus.cc:2468] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:06.183682 30716 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.168444 31541 raft_consensus.cc:697] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 6 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:06.204016 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.205542 30598 raft_consensus.cc:2468] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:06.207358 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.208287 30598 raft_consensus.cc:2468] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:06.188289 31209 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:06.209551 31532 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:06.210423 31607 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:06.200008 30596 raft_consensus.cc:3060] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:06.173846 31485 raft_consensus.cc:2764] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251025 20:04:06.201715 30716 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:06.212481 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 112911us
I20251025 20:04:06.226709 30719 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:06.177031 31563 raft_consensus.cc:697] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:06.188598 31423 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:06.230195 31423 raft_consensus.cc:3060] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:06.231331 31614 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:06.224334 30744 raft_consensus.cc:2393] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:06.172500 31604 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:06.234117 31604 raft_consensus.cc:3060] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:06.217105 30678 raft_consensus.cc:2468] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:06.226838 30674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 493) took 1031 ms. Trace:
I20251025 20:04:06.224761 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.239276 30674 rpcz_store.cc:276] 1025 20:04:05.195293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.002712 (+807419us) service_pool.cc:225] Handling call
1025 20:04:06.026816 (+ 24104us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:06.027062 (+ 246us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.027089 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:06.027569 (+ 480us) log.cc:844] Serialized 26 byte log entry
1025 20:04:06.112959 (+ 85390us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.113021 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.113131 (+ 110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:06.226582 (+113451us) raft_consensus.cc:1685] finished
1025 20:04:06.226604 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.226769 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":84824,"threads_started":1,"wal-append.queue_time_us":142916}
I20251025 20:04:06.253866 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.255684 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.256510 30598 raft_consensus.cc:2468] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
W20251025 20:04:06.259918 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 58335.6us
I20251025 20:04:06.298053 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:06.298462 30716 leader_election.cc:304] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:06.264621 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 148222us
I20251025 20:04:06.319507 30645 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.320775 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.319873 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 478) took 2475 ms. Trace:
I20251025 20:04:06.173980 30595 rpcz_store.cc:276] 1025 20:04:04.034679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.077369 (+2042690us) service_pool.cc:225] Handling call
1025 20:04:06.077748 (+ 379us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.098076 (+ 20328us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.099071 (+ 995us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.106018 (+ 6947us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.110796 (+ 4778us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.117762 (+ 6966us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.149058 (+ 31296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.278755 31614 raft_consensus.cc:2749] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:06.296684 30671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 496) took 1081 ms. Trace:
I20251025 20:04:06.282969 31549 raft_consensus.cc:2804] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:06.324523 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:06.302011 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 201749us
I20251025 20:04:06.260274 31606 raft_consensus.cc:697] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:06.298995 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 72397.1us
I20251025 20:04:06.321480 30602 rpcz_store.cc:276] 1025 20:04:03.844944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.025583 (+2180639us) service_pool.cc:225] Handling call
1025 20:04:06.026059 (+ 476us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.026267 (+ 208us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:06.154677 (+128410us) log.cc:844] Serialized 14 byte log entry
1025 20:04:06.319482 (+164805us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.319553 (+ 71us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:06.319600 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.319689 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.319813 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":176932,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":50965,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":16019,"spinlock_wait_cycles":2483072,"thread_start_us":281720,"threads_started":3,"wal-append.queue_time_us":123043}
I20251025 20:04:06.297806 31268 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:06.279213 31607 raft_consensus.cc:697] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:06.326594 31625 raft_consensus.cc:2804] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:04:06.326594 30598 raft_consensus.cc:2393] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:06.337584 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 486) took 2258 ms. Trace:
I20251025 20:04:06.339787 31630 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:06.340436 31630 raft_consensus.cc:697] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:06.340150 31629 consensus_queue.cc:1048] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:06.341503 31630 consensus_queue.cc:237] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.342322 31414 leader_election.cc:290] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:06.338194 30645 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.252476 31532 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:06.339109 30602 rpcz_store.cc:276] 1025 20:04:04.078807 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.327511 (+2248704us) service_pool.cc:225] Handling call
1025 20:04:06.337000 (+ 9489us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.337221 (+ 221us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.337255 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.337305 (+ 50us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.337339 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.337398 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.337516 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:06.332715 31622 log.cc:927] Time spent T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20251025 20:04:06.327345 31549 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:06.344973 30645 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.345150 30567 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:06.353987 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.356182 30678 raft_consensus.cc:3060] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:06.356055 30645 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.361706 30750 raft_consensus.cc:1240] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:06.331265 30596 raft_consensus.cc:2468] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:06.364609 30678 raft_consensus.cc:2468] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:06.365520 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.365787 30716 leader_election.cc:304] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.366281 30678 raft_consensus.cc:2393] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:06.348067 31508 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:06.369163 30567 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:06.348053 31625 raft_consensus.cc:697] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:06.318158 31563 consensus_queue.cc:237] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.370994 30567 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:06.371623 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.304088 31541 consensus_queue.cc:237] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.367960 31614 raft_consensus.cc:2804] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:04:06.374979 30678 raft_consensus.cc:3060] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:06.339284 31268 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.347764 31290 consensus_queue.cc:1048] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20251025 20:04:06.357336 31549 raft_consensus.cc:3060] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:06.367329 31615 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:06.346843 31604 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.368552 31318 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:06.354698 31532 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:06.394788 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:06.374560 31560 raft_consensus.cc:2804] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:06.444406 30596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 513) took 1973 ms. Trace:
W20251025 20:04:06.443877 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 70641.1us
I20251025 20:04:06.374758 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:06.373056 31508 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:06.372409 31628 raft_consensus.cc:2749] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:06.450477 30678 raft_consensus.cc:2468] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:06.380378 31607 consensus_queue.cc:237] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.185719 30675 rpcz_store.cc:276] 1025 20:04:05.184419 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.997017 (+812598us) service_pool.cc:225] Handling call
1025 20:04:05.997281 (+ 264us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:05.997473 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:06.062036 (+ 64563us) log.cc:844] Serialized 14 byte log entry
1025 20:04:06.183117 (+121081us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.183177 (+ 60us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:06.183225 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.183310 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.183457 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15961,"prepare.queue_time_us":45914,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":4688,"thread_start_us":115996,"threads_started":1}
W20251025 20:04:06.559787 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 188863us
I20251025 20:04:06.479214 30602 raft_consensus.cc:3060] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
W20251025 20:04:06.586589 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:06.458105 30716 leader_election.cc:336] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Timed out: RequestConsensusVote RPC to 127.28.192.194:32975 timed out after 1.962s (SENT)
I20251025 20:04:06.439484 31632 raft_consensus.cc:2804] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:06.645669 30567 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:06.646705 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 188685us
I20251025 20:04:06.327361 30671 rpcz_store.cc:276] 1025 20:04:05.215153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.024445 (+809292us) service_pool.cc:225] Handling call
1025 20:04:06.024747 (+ 302us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.025065 (+ 318us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:06.172099 (+147034us) log.cc:844] Serialized 14 byte log entry
1025 20:04:06.270998 (+ 98899us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.280231 (+ 9233us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:06.296291 (+ 16060us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.296411 (+ 120us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.296604 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47895,"prepare.queue_time_us":48116,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":5411,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":16114,"thread_start_us":149879,"threads_started":2,"wal-append.queue_time_us":168740}
I20251025 20:04:06.386242 31614 raft_consensus.cc:697] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:06.388053 30595 raft_consensus.cc:1275] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251025 20:04:06.672650 30716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:06.441732 31560 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:06.673568 30716 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:06.674045 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.676923 30675 raft_consensus.cc:2393] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
W20251025 20:04:06.636691 31634 log.cc:927] Time spent T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20251025 20:04:06.679055 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 509) took 2305 ms. Trace:
I20251025 20:04:06.679756 30675 raft_consensus.cc:1240] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:06.679602 30598 rpcz_store.cc:276] 1025 20:04:04.373465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.339415 (+1965950us) service_pool.cc:225] Handling call
1025 20:04:06.347185 (+ 7770us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:06.348065 (+ 880us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.348099 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:06.348578 (+ 479us) log.cc:844] Serialized 26 byte log entry
1025 20:04:06.644472 (+295894us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.644535 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.644620 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:06.678750 (+ 34130us) raft_consensus.cc:1685] finished
1025 20:04:06.678782 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.678969 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7966,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":16110,"thread_start_us":119490,"threads_started":1,"wal-append.queue_time_us":321580}
I20251025 20:04:06.682482 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 516) took 2143 ms. Trace:
I20251025 20:04:06.682861 30598 rpcz_store.cc:276] 1025 20:04:04.539469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.681654 (+2142185us) service_pool.cc:225] Handling call
1025 20:04:06.681988 (+ 334us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.682173 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.682204 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.682243 (+ 39us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:06.682275 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.682327 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.682424 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.685103 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 391) took 2127 ms. Trace:
W20251025 20:04:06.505964 31567 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.070s user 0.023s sys 0.021s
W20251025 20:04:06.516966 31633 log.cc:927] Time spent T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20251025 20:04:06.685628 30598 rpcz_store.cc:276] 1025 20:04:04.557322 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.684100 (+2126778us) service_pool.cc:225] Handling call
1025 20:04:06.684458 (+ 358us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.684676 (+ 218us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.684717 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.684778 (+ 61us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.684823 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.684895 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.685022 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.637466 31209 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.637699 31268 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:06.494765 31192 consensus_queue.cc:1059] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.202s
I20251025 20:04:06.451756 31508 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:06.494324 31615 raft_consensus.cc:697] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 5 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:06.731786 31645 raft_consensus.cc:2804] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:06.565414 31318 raft_consensus.cc:697] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:06.518728 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:06.483722 30748 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:06.732872 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 59372.5us
I20251025 20:04:06.734676 30598 raft_consensus.cc:1240] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:06.697757 30800 tablet_bootstrap.cc:654] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:06.749872 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 392) took 2189 ms. Trace:
I20251025 20:04:06.693233 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:06.736944 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 409848us
I20251025 20:04:06.559216 31423 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.752959 30673 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:06.753280 30671 raft_consensus.cc:2393] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:06.753341 30719 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:06.754105 30745 raft_consensus.cc:1240] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:06.754735 30719 leader_election.cc:400] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 5. Current term is 6.
I20251025 20:04:06.755064 30719 leader_election.cc:403] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:04:06.756091 30719 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:06.696327 30595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 492) took 2584 ms. Trace:
I20251025 20:04:06.671340 31629 raft_consensus.cc:2749] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251025 20:04:06.695855 30677 consensus_meta.cc:287] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.515s user 0.010s sys 0.002s
W20251025 20:04:06.738061 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 289867us
W20251025 20:04:06.744931 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 79411.5us
I20251025 20:04:06.761431 30567 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:06.762660 30567 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:06.684757 31560 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:06.672456 31632 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:06.750278 30598 rpcz_store.cc:276] 1025 20:04:04.560860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.731192 (+2170332us) service_pool.cc:225] Handling call
1025 20:04:06.733870 (+ 2678us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.749568 (+ 15698us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.749803 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.770705 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 517) took 2199 ms. Trace:
I20251025 20:04:06.771106 30598 rpcz_store.cc:276] 1025 20:04:04.571080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.769847 (+2198767us) service_pool.cc:225] Handling call
1025 20:04:06.770220 (+ 373us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.770392 (+ 172us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.770417 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.770456 (+ 39us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.770484 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.770537 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.770646 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.748947 31645 raft_consensus.cc:697] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:06.775764 30673 raft_consensus.cc:2410] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:06.775259 31645 consensus_queue.cc:237] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:06.776337 30673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 310) took 1451 ms (client timeout 1601 ms). Trace:
W20251025 20:04:06.776719 30673 rpcz_store.cc:269] 1025 20:04:05.325076 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.333612 (+ 8536us) service_pool.cc:225] Handling call
1025 20:04:06.518530 (+1184918us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea507ce 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:06.776270 (+257740us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47971712}
I20251025 20:04:05.235602 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:06.765215 31628 raft_consensus.cc:2749] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:06.772099 31632 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:06.751842 31318 consensus_queue.cc:237] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.692396 31532 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:06.387349 31606 consensus_queue.cc:237] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:06.746434 31567 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.310s user 0.043s sys 0.058s
I20251025 20:04:06.757254 30719 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:06.789885 30719 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:06.764443 30567 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:06.766211 31644 raft_consensus.cc:2804] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:06.766531 31268 raft_consensus.cc:2749] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:06.447606 30596 rpcz_store.cc:276] 1025 20:04:04.470842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.378575 (+1907733us) service_pool.cc:225] Handling call
1025 20:04:06.443713 (+ 65138us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.443934 (+ 221us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.444044 (+ 110us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.444095 (+ 51us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.444127 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.444194 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.444333 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.793615 30567 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:06.796147 30567 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:06.763363 30677 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:06.801653 30679 raft_consensus.cc:1240] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:06.764979 31635 raft_consensus.cc:2749] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 5. Current term is 6.
I20251025 20:04:06.777820 30599 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:06.814203 31209 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:06.794703 31644 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:06.760351 30595 rpcz_store.cc:276] 1025 20:04:04.112187 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.344455 (+2232268us) service_pool.cc:225] Handling call
1025 20:04:06.365222 (+ 20767us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.675514 (+310292us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.696235 (+ 20721us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:06.799165 31641 raft_consensus.cc:2804] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:06.797631 31532 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:06.820648 31629 raft_consensus.cc:2749] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:06.522372 31625 consensus_queue.cc:237] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.772142 31614 consensus_queue.cc:237] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:06.850116 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 59080.1us
W20251025 20:04:06.835651 30599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 494) took 2705 ms (client timeout 1852 ms). Trace:
I20251025 20:04:07.411146 30719 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:07.412431 31610 raft_consensus.cc:2804] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:07.413015 31610 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:07.413380 31610 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:06.800761 31628 raft_consensus.cc:2804] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:06.822011 31607 raft_consensus.cc:2804] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:06.859077 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 394) took 2282 ms. Trace:
I20251025 20:04:06.854468 31641 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:07.414328 31628 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:06.849104 31644 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.414433 31607 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:07.414822 31628 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.415100 31607 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:06.823683 31615 consensus_queue.cc:237] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.920281 31508 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:06.853410 31532 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:07.442807 31532 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:07.443549 31508 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:06.836333 31423 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:06.898886 31560 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:07.086679 31656 log.cc:927] Time spent T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.162s user 0.000s sys 0.000s
I20251025 20:04:07.087641 31632 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:07.101318 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 305212us
I20251025 20:04:07.489429 31560 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.490129 30567 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.490432 31632 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.101881 30596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 518) took 2492 ms. Trace:
I20251025 20:04:07.491035 31632 raft_consensus.cc:2804] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20251025 20:04:07.190201 30604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 505) took 2997 ms (client timeout 1592 ms). Trace:
I20251025 20:04:07.491158 30567 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.491559 31632 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:07.194516 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.492107 31632 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.492219 30567 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:07.490963 30596 rpcz_store.cc:276] 1025 20:04:04.609241 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.818078 (+2208837us) service_pool.cc:225] Handling call
1025 20:04:06.864174 (+ 46096us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:06.908775 (+ 44601us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.908826 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:06.909591 (+ 765us) log.cc:844] Serialized 26 byte log entry
1025 20:04:07.079002 (+169411us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:07.079449 (+ 447us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:07.079595 (+ 146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:07.101571 (+ 21976us) raft_consensus.cc:1685] finished
1025 20:04:07.101598 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:07.101799 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":220431,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":156,"thread_start_us":136558,"threads_started":1,"wal-append.queue_time_us":188909}
W20251025 20:04:07.492835 30603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 535) took 1397 ms (client timeout 1851 ms). Trace:
I20251025 20:04:07.493124 31530 raft_consensus.cc:2804] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20251025 20:04:07.197675 31647 log.cc:927] Time spent T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.295s user 0.000s sys 0.000s
I20251025 20:04:07.493805 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:07.207027 30570 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34886 dropped due to backpressure: service queue is full with 50 items [suppressed 7 similar messages]
I20251025 20:04:07.494526 30596 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:07.494602 30567 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.495438 31629 raft_consensus.cc:2804] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:07.495843 31629 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:07.496204 31629 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:07.492118 31560 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:07.497435 31560 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:07.497792 31560 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:07.498363 30678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 502) took 2018 ms. Trace:
I20251025 20:04:07.498878 30678 rpcz_store.cc:276] 1025 20:04:05.480221 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.647858 (+1167637us) service_pool.cc:225] Handling call
1025 20:04:06.657704 (+ 9846us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:06.658207 (+ 503us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.658247 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:06.670089 (+ 11842us) log.cc:844] Serialized 26 byte log entry
1025 20:04:06.787779 (+117690us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.790518 (+ 2739us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.790636 (+ 118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:07.196946 (+406310us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:07.497971 (+301025us) raft_consensus.cc:1685] finished
1025 20:04:07.497998 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:07.498292 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9667,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":135,"spinlock_wait_cycles":7303936,"thread_start_us":117160,"threads_started":1,"wal-append.queue_time_us":179578}
W20251025 20:04:07.212944 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:07.501312 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:07.501785 31632 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.222620 31639 raft_consensus.cc:3055] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:07.233563 30600 raft_consensus.cc:2393] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.502804 31639 raft_consensus.cc:740] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:07.503810 31639 consensus_queue.cc:260] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:07.252916 31652 log.cc:927] Time spent T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.369s user 0.001s sys 0.000s
I20251025 20:04:07.264627 30744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 467) took 1230 ms. Trace:
I20251025 20:04:07.505340 30744 rpcz_store.cc:276] 1025 20:04:06.034177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.060155 (+ 25978us) service_pool.cc:225] Handling call
1025 20:04:07.264287 (+1204132us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b540020bf54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:07.264524 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":269763968}
I20251025 20:04:07.271859 30675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 508) took 1234 ms. Trace:
W20251025 20:04:07.278648 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 455822us
W20251025 20:04:07.283486 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:04:07.295637 30601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 465) took 4186 ms. Trace:
I20251025 20:04:07.327708 30800 tablet_bootstrap.cc:492] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:04:07.507628 30800 ts_tablet_manager.cc:1403] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 1.407s user 0.022s sys 0.008s
I20251025 20:04:07.507279 30601 rpcz_store.cc:276] 1025 20:04:03.108874 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:03.164292 (+ 55418us) service_pool.cc:225] Handling call
1025 20:04:03.999497 (+835205us) spinlock_profiling.cc:243] Waited 9.99 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:03.999896 (+ 399us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:04.079971 (+ 80075us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:04.083248 (+ 3277us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:04.090122 (+ 6874us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:04.093877 (+ 3755us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:07.295326 (+3201449us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b54000406d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:07.295376 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:07.295552 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93696000}
I20251025 20:04:07.509387 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.510763 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.511091 30800 raft_consensus.cc:359] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.511876 30800 raft_consensus.cc:385] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 20:04:07.335441 30643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.011s sys 0.019s
I20251025 20:04:07.512049 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:07.512653 30643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.703s user 0.012s sys 0.019s
I20251025 20:04:07.350131 30674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 336) took 1984 ms. Trace:
I20251025 20:04:07.512898 30601 raft_consensus.cc:2468] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:07.506613 30675 rpcz_store.cc:276] 1025 20:04:06.037786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.681136 (+643350us) service_pool.cc:225] Handling call
1025 20:04:06.681398 (+ 262us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.681561 (+ 163us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:06.707527 (+ 25966us) log.cc:844] Serialized 14 byte log entry
1025 20:04:07.252255 (+544728us) spinlock_profiling.cc:243] Waited 7.35 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:07.271374 (+ 19119us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:07.271447 (+ 73us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:07.271511 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:07.271596 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:07.271774 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9799,"prepare.queue_time_us":25416,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":62661,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":10716,"spinlock_wait_cycles":16895360,"thread_start_us":553416,"threads_started":2,"wal-append.queue_time_us":150774}
I20251025 20:04:07.514110 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:07.493227 30603 rpcz_store.cc:269] 1025 20:04:06.095855 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.135722 (+ 39867us) service_pool.cc:225] Handling call
1025 20:04:07.194295 (+1058573us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:07.492758 (+298463us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5237504}
I20251025 20:04:07.513198 30674 rpcz_store.cc:276] 1025 20:04:05.365382 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.241074 (+875692us) service_pool.cc:225] Handling call
1025 20:04:06.298112 (+ 57038us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:06.298561 (+ 449us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.298592 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:06.299323 (+ 731us) log.cc:844] Serialized 26 byte log entry
1025 20:04:06.345557 (+ 46234us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:06.346063 (+ 506us) log.cc:844] Serialized 14 byte log entry
1025 20:04:06.346788 (+ 725us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.346867 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:07.349817 (+1002950us) raft_consensus.cc:1685] finished
1025 20:04:07.349843 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:07.350057 (+ 214us) inbound_call.cc:173] Queueing success response
1025 20:04:07.429811 (+ 79754us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b18002de320. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8278d09a1 00007fd8278dd371 00007fd81ea8d992 00007fd81ea8fd61 00007fd81ea8fd19 00007fd81ea8fca9 00007fd81ea8fc71 00007fd81ea8ef6d 0000000000379254 0000000000379099 00007fd81ea70d1a
Metrics: {"mutex_wait_us":1237,"prepare.queue_time_us":174,"prepare.run_cpu_time_us":203,"prepare.run_wall_time_us":1592,"raft.queue_time_us":185,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":19445,"spinlock_wait_cycles":4646656,"thread_start_us":45789,"threads_started":1,"wal-append.queue_time_us":71011}
I20251025 20:04:07.352537 30749 raft_consensus.cc:1240] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:07.514743 30601 raft_consensus.cc:2468] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:07.515450 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:07.354255 30716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.013s sys 0.009s
W20251025 20:04:07.516177 30716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.759s user 0.013s sys 0.009s
W20251025 20:04:07.363194 30748 consensus_meta.cc:287] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.611s user 0.015s sys 0.000s
I20251025 20:04:07.363323 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:07.365816 30597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 511) took 2933 ms (client timeout 1526 ms). Trace:
W20251025 20:04:07.374006 31549 consensus_meta.cc:287] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.935s user 0.015s sys 0.007s
I20251025 20:04:07.379181 31612 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:07.384649 30745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 479) took 1024 ms. Trace:
I20251025 20:04:07.391568 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:07.394794 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:07.395790 30679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 330) took 2039 ms. Trace:
I20251025 20:04:07.414458 30598 rpcz_store.cc:276] 1025 20:04:04.576386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.772017 (+2195631us) service_pool.cc:225] Handling call
1025 20:04:06.772327 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.772519 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:06.785107 (+ 12588us) log.cc:844] Serialized 14 byte log entry
1025 20:04:06.858312 (+ 73205us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:06.858404 (+ 92us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:06.858446 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:06.858502 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:06.858672 (+ 170us) inbound_call.cc:173] Queueing success response
1025 20:04:07.334347 (+475675us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b1800368d40. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8278d09a1 00007fd8278dd371 00007fd81ea8d992 00007fd81ea8fd61 00007fd81ea8fd19 00007fd81ea8fca9 00007fd81ea8fc71 00007fd81ea8ef6d 0000000000379254 0000000000379099 00007fd81ea70d1a
Metrics: {"prepare.queue_time_us":46177,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":72,"raft.queue_time_us":8961,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":127,"spinlock_wait_cycles":3115904,"thread_start_us":83385,"threads_started":2,"wal-append.queue_time_us":76023}
I20251025 20:04:07.419637 30719 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
W20251025 20:04:07.409536 30599 rpcz_store.cc:269] 1025 20:04:04.118342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.235246 (+1116904us) service_pool.cc:225] Handling call
1025 20:04:06.821075 (+1585829us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:07.414584 31641 raft_consensus.cc:3060] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.428787 30746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 473) took 1070 ms. Trace:
W20251025 20:04:07.447116 31617 log.cc:927] Time spent T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 1.518s user 0.007s sys 0.014s
I20251025 20:04:07.449438 31645 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:07.451200 31567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.001s sys 0.009s
I20251025 20:04:07.457657 31604 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:07.457657 31532 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.462750 31644 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.494872 30570 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34886 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 459)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 460)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 461)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 443)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 421)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 463)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 429)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 465)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 457)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 426)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 456)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 458)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 438)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 427)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 436)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 447)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 445)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 440)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 442)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 448)
I20251025 20:04:07.469834 31610 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.473735 31628 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.475427 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 512) took 3036 ms. Trace:
I20251025 20:04:07.475749 31607 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.493603 31530 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:07.504869 30600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 502) took 3314 ms (client timeout 1739 ms). Trace:
I20251025 20:04:07.516357 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.630296 31612 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.631260 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.631413 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.632256 30599 raft_consensus.cc:2393] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
W20251025 20:04:07.632375 31567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.795s user 0.002s sys 0.025s
I20251025 20:04:07.632802 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.633369 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.652453 30599 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:07.658069 30500 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39186 (request call id 98) took 7895 ms. Trace:
I20251025 20:04:07.634893 31478 raft_consensus.cc:2804] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251025 20:04:07.635627 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 216059us
I20251025 20:04:07.663753 31478 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:07.664265 31478 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:07.664136 30719 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:07.665060 31604 raft_consensus.cc:2804] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:07.665431 31604 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:04:07.665467 30719 leader_election.cc:343] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:04:07.665869 31604 raft_consensus.cc:3060] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.667073 30719 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:07.667870 31499 raft_consensus.cc:2804] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:07.668234 31499 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:07.668574 31499 raft_consensus.cc:3060] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.634904 31549 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.636168 30745 rpcz_store.cc:276] 1025 20:04:06.360573 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.360694 (+ 121us) service_pool.cc:225] Handling call
1025 20:04:06.370803 (+ 10109us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.754473 (+383670us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:07.384289 (+629816us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54001441d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:07.384560 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":881042176}
I20251025 20:04:07.635722 30679 rpcz_store.cc:276] 1025 20:04:05.356265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.199172 (+842907us) service_pool.cc:225] Handling call
1025 20:04:06.801386 (+602214us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:06.802201 (+ 815us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:07.393737 (+591536us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b540018ddd0. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3944d 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:07.395687 (+ 1950us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1383804288}
I20251025 20:04:07.634979 30746 rpcz_store.cc:276] 1025 20:04:06.358509 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.365450 (+ 6941us) service_pool.cc:225] Handling call
1025 20:04:07.423541 (+1058091us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:07.424425 (+ 884us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:07.427136 (+ 2711us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:07.427653 (+ 517us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:07.428189 (+ 536us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:07.428458 (+ 269us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:07.428540 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:07.428721 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8766336}
I20251025 20:04:07.506793 31632 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.504925 31639 raft_consensus.cc:3060] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.670464 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.670789 31549 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:07.638846 31530 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.674964 30719 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:07.658420 31532 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.639470 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.679005 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.680256 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.681353 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.681087 31641 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.639977 30595 raft_consensus.cc:1240] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:07.681458 31549 consensus_queue.cc:1048] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:07.682348 30747 raft_consensus.cc:1240] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20251025 20:04:07.684154 30595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 522) took 1592 ms. Trace:
I20251025 20:04:07.684566 30595 rpcz_store.cc:276] 1025 20:04:06.092251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.862588 (+770337us) service_pool.cc:225] Handling call
1025 20:04:06.881489 (+ 18901us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:07.682086 (+800597us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:07.683907 (+ 1821us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b54003084d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:07.684094 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1695724416}
W20251025 20:04:07.639490 30670 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 317) took 2312 ms (client timeout 1725 ms). Trace:
I20251025 20:04:07.685864 31478 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.516623 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:07.546109 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 51575.1us
I20251025 20:04:07.687805 30747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 478) took 1327 ms. Trace:
I20251025 20:04:07.688252 30747 rpcz_store.cc:276] 1025 20:04:06.360269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.364445 (+ 4176us) service_pool.cc:225] Handling call
1025 20:04:07.297050 (+932605us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:07.297457 (+ 407us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:07.682749 (+385292us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:07.687511 (+ 4762us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b54002a1a54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:07.687740 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":892762752}
I20251025 20:04:07.638870 30672 raft_consensus.cc:2376] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4. Re-sending same reply.
W20251025 20:04:07.690034 30672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 481) took 2615 ms (client timeout 1962 ms). Trace:
W20251025 20:04:07.690410 30672 rpcz_store.cc:269] 1025 20:04:05.074668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.149739 (+ 75071us) service_pool.cc:225] Handling call
1025 20:04:07.394525 (+2244786us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:07.689946 (+295421us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3367808}
I20251025 20:04:07.561378 30744 raft_consensus.cc:1275] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 20:04:07.512172 30800 raft_consensus.cc:740] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:04:07.691933 30800 consensus_queue.cc:260] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:07.535822 30645 leader_election.cc:336] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.516s (SENT)
I20251025 20:04:07.695716 30645 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:07.696646 31290 raft_consensus.cc:2749] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251025 20:04:07.642220 30751 raft_consensus.cc:1240] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3. Ops: []
W20251025 20:04:07.639245 30597 rpcz_store.cc:269] 1025 20:04:04.432521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:07.365615 (+2933094us) service_pool.cc:212] Skipping call since client already timed out
1025 20:04:07.365730 (+ 115us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251025 20:04:07.693782 30800 ts_tablet_manager.cc:1434] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.186s user 0.005s sys 0.005s
I20251025 20:04:07.698247 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.699569 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.700271 30597 raft_consensus.cc:2468] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
F20251025 20:04:07.701218 29443 test_workload.cc:382] Timed out: Timed out waiting for Table Creation
I20251025 20:04:07.515847 30749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 477) took 1155 ms. Trace:
I20251025 20:04:07.643777 30748 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
W20251025 20:04:07.516517 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 759771us
I20251025 20:04:07.538460 30750 raft_consensus.cc:1240] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:07.646348 30596 raft_consensus.cc:2410] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:07.647200 31560 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.648312 31629 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.621577 30501 catalog_manager.cc:5649] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 2, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:07.655606 31628 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.656446 31610 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:07.491729 30604 rpcz_store.cc:269] 1025 20:04:04.192638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.282889 (+ 90251us) service_pool.cc:225] Handling call
1025 20:04:07.189848 (+2906959us) spinlock_profiling.cc:243] Waited 88.8 ms on lock 0x7b54001f8454. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:07.190092 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":204353408}
I20251025 20:04:07.657320 31607 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:07.633646 30601 raft_consensus.cc:2393] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.658677 30753 raft_consensus.cc:1240] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:07.634397 31644 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.658443 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.632858 31612 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.633884 30602 rpcz_store.cc:276] 1025 20:04:04.438900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.350766 (+1911866us) service_pool.cc:225] Handling call
1025 20:04:06.351028 (+ 262us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:06.795136 (+444108us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:06.795168 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:06.795682 (+ 514us) log.cc:844] Serialized 26 byte log entry
1025 20:04:07.389082 (+593400us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b1800050800. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8278d09a1 00007fd8278dd371 00007fd81ea702f7 00007fd827849219 00007fd81ea8cc4b 00007fd81ea8aeef 00007fd81ea8d1b2 00007fd8216d42bb 00007fd8216d9dc3 00007fd8216da171 00007fd821af565f
1025 20:04:07.393131 (+ 4049us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:07.404625 (+ 11494us) log.cc:844] Serialized 14 byte log entry
1025 20:04:07.406102 (+ 1477us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:07.473735 (+ 67633us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54001234d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:07.473795 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:07.473845 (+ 50us) raft_consensus.cc:1685] finished
1025 20:04:07.473859 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:07.475357 (+ 1498us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2098,"prepare.queue_time_us":10912,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":2490,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":14083,"spinlock_wait_cycles":298449536,"thread_start_us":14893,"threads_started":2,"wal-append.queue_time_us":597446}
I20251025 20:04:07.719271 30749 rpcz_store.cc:276] 1025 20:04:06.359913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.365470 (+ 5557us) service_pool.cc:225] Handling call
1025 20:04:07.351957 (+986487us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:07.352254 (+ 297us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:07.515507 (+163253us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:07.515779 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9792768}
I20251025 20:04:07.727293 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:07.686059 30670 rpcz_store.cc:269] 1025 20:04:05.327176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.359104 (+ 31928us) service_pool.cc:225] Handling call
1025 20:04:07.391324 (+2032220us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:07.639391 (+248067us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":79476096}
I20251025 20:04:07.683765 31499 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.546623 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:07.728631 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.729086 30752 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:07.729399 30602 raft_consensus.cc:2393] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.717362 30500 rpcz_store.cc:276] 1025 20:03:59.762398 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:03:59.767274 (+ 4876us) service_pool.cc:225] Handling call
1025 20:04:07.658000 (+7890726us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 20:04:04.724832 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 20:04:04.729731 (+ 4899us) write_op.cc:432] Acquiring schema lock in shared mode
1025 20:04:04.729774 (+ 43us) write_op.cc:435] Acquired schema lock
1025 20:04:04.729800 (+ 26us) tablet.cc:660] Decoding operations
1025 20:04:05.075448 (+345648us) write_op.cc:620] Acquiring the partition lock for write op
1025 20:04:05.095687 (+ 20239us) write_op.cc:641] Partition lock acquired for write op
1025 20:04:05.097434 (+ 1747us) tablet.cc:683] Acquiring locks for 16 operations
1025 20:04:05.203288 (+105854us) tablet.cc:699] Row locks acquired
1025 20:04:05.204484 (+ 1196us) write_op.cc:260] PREPARE: finished
1025 20:04:05.213339 (+ 8855us) write_op.cc:270] Start()
1025 20:04:05.219805 (+ 6466us) write_op.cc:276] Timestamp: P: 1761422645213264 usec, L: 0
1025 20:04:05.221336 (+ 1531us) op_driver.cc:348] REPLICATION: starting
1025 20:04:05.318558 (+ 97222us) log.cc:844] Serialized 6401 byte log entry
1025 20:04:05.505737 (+187179us) op_driver.cc:464] REPLICATION: finished
1025 20:04:05.541498 (+ 35761us) write_op.cc:301] APPLY: starting
1025 20:04:05.541591 (+ 93us) tablet.cc:1365] starting BulkCheckPresence
1025 20:04:05.541942 (+ 351us) tablet.cc:1368] finished BulkCheckPresence
1025 20:04:05.642256 (+100314us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 20:04:05.850996 (+208740us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 20:04:05.868213 (+ 17217us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
1025 20:04:05.869833 (+ 1620us) write_op.cc:312] APPLY: finished
1025 20:04:05.890976 (+ 21143us) log.cc:844] Serialized 112 byte log entry
1025 20:04:05.892613 (+ 1637us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:04:05.898055 (+ 5442us) write_op.cc:454] Released schema lock
1025 20:04:05.898123 (+ 68us) write_op.cc:341] FINISH: Updating metrics
1025 20:04:05.898531 (+ 408us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:04:05.898562 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":35365,"apply.run_cpu_time_us":8060,"apply.run_wall_time_us":356980,"num_ops":16,"prepare.queue_time_us":419428,"prepare.run_cpu_time_us":34335,"prepare.run_wall_time_us":704601,"raft.queue_time_us":72152,"raft.run_cpu_time_us":11394,"raft.run_wall_time_us":41928,"replication_time_us":283015,"thread_start_us":374484,"threads_started":4,"wal-append.queue_time_us":98761,"wal-append.run_cpu_time_us":11401,"wal-append.run_wall_time_us":1477932}]]}
I20251025 20:04:07.701181 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.731535 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.732630 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.733779 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.733986 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:07.642270 30600 rpcz_store.cc:269] 1025 20:04:04.190214 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:04.595445 (+405231us) service_pool.cc:225] Handling call
1025 20:04:07.504618 (+2909173us) spinlock_profiling.cc:243] Waited 2.63 s on lock 0x7b54000523d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:07.504793 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6060460288}
I20251025 20:04:07.687767 30603 raft_consensus.cc:2393] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.721148 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.735512 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.735693 30596 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:04:07.735744 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.736532 30603 raft_consensus.cc:2368] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3.
I20251025 20:04:07.736652 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.736989 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.737298 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.737145 31530 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
*** Check failure stack trace: ***
*** Aborted at 1761422647 (unix time) try "date -d @1761422647" if you are using GNU date ***
I20251025 20:04:07.697412 30719 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:04:07.763898 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 66572.3us
I20251025 20:04:07.766942 31428 raft_consensus.cc:2749] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:07.767938 31530 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.767794 31428 consensus_queue.cc:1059] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:04:07.769352 31428 raft_consensus.cc:3055] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:07.703768 31641 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:07.694422 31604 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.638492 31645 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.769762 31428 raft_consensus.cc:740] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:07.777593 31428 consensus_queue.cc:260] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.715058 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.716028 30676 raft_consensus.cc:2468] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
W20251025 20:04:07.780474 30676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 315) took 2453 ms (client timeout 1680 ms). Trace:
W20251025 20:04:07.780841 30676 rpcz_store.cc:269] 1025 20:04:05.326595 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:05.358185 (+ 31590us) service_pool.cc:225] Handling call
1025 20:04:07.363109 (+2004924us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:07.780287 (+417178us) spinlock_profiling.cc:243] Waited 75.4 ms on lock 0x7b54001577d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:07.780404 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":205040384}
I20251025 20:04:07.716710 31478 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:07.777773 30568 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34874 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 555)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 546)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 436)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 536)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 471)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 447)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 474)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 477)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 545)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 478)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 480)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 544)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 445)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 440)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 442)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 448)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 481)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 543)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 547)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 552)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 554)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 556)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 557)
I20251025 20:04:07.689363 30567 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.738263 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:07.783681 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 94408.4us
I20251025 20:04:07.783993 31428 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:07.784349 30567 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.784555 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:07.782660 30568 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34874 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 555)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 546)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 436)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 536)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 471)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 447)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 474)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 477)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 545)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 478)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 480)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 544)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 445)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 440)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 442)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 448)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 481)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 543)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 547)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 552)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 554)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 556)
I20251025 20:04:07.785367 30567 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.719590 30748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 474) took 1360 ms. Trace:
I20251025 20:04:07.739085 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.722218 31414 raft_consensus.cc:2764] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251025 20:04:07.724129 30716 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:07.785916 30748 rpcz_store.cc:276] 1025 20:04:06.358969 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:06.364323 (+ 5354us) service_pool.cc:225] Handling call
1025 20:04:07.719529 (+1355206us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:07.786926 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 62888.7us
I20251025 20:04:07.787009 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:07.787590 31478 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:07.788164 31478 raft_consensus.cc:697] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:07.788198 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:07.789451 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:07.725711 31629 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.721882 31560 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.726519 30599 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:07.790438 31610 consensus_queue.cc:1048] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:04:07.791247 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.740496 30603 raft_consensus.cc:2393] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.791886 30599 raft_consensus.cc:2468] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:07.726518 30501 catalog_manager.cc:5649] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 2, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:07.792802 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.793200 30567 leader_election.cc:304] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.793088 30501 catalog_manager.cc:5649] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 2, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:07.721290 31491 raft_consensus.cc:2804] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:07.794056 31532 raft_consensus.cc:2804] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:07.794129 30567 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.794577 31491 raft_consensus.cc:697] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:07.794860 31632 raft_consensus.cc:2804] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:07.789284 31478 consensus_queue.cc:237] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.795346 31632 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:07.794757 30501 catalog_manager.cc:5649] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:07.795800 31632 raft_consensus.cc:3060] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.795609 31491 consensus_queue.cc:237] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.726061 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.797668 30601 raft_consensus.cc:2393] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.719478 30800 tablet_bootstrap.cc:492] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:04:07.737310 30600 raft_consensus.cc:2468] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:07.745196 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.799629 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:07.800696 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.734968 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.737638 30753 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:04:07.746116 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.725863 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.746152 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.752274 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.753305 31499 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:07.750890 30752 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
W20251025 20:04:07.696792 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 161035us
W20251025 20:04:07.817740 30570 connection.cc:232] RPC call timeout handler was delayed by -2.6623772119999103s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1877s for a response
W20251025 20:04:07.818418 30645 leader_election.cc:336] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.517s (SENT)
I20251025 20:04:07.818691 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:07.819327 30645 leader_election.cc:336] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.550s (SENT)
I20251025 20:04:07.771807 31604 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:07.771852 31645 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:07.685813 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.818681 30570 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:07.821676 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.782503 31607 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:07.785539 30744 raft_consensus.cc:2393] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:07.736667 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.739887 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.832899 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.792496 30568 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34874 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 436)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 536)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 471)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 447)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 474)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 477)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 545)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 478)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 480)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 544)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 445)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 564)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 440)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 442)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 448)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 481)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 543)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 547)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 552)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 554)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 556)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 562)
I20251025 20:04:07.804620 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:07.808910 30674 raft_consensus.cc:1275] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:07.801188 31612 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:07.794504 31532 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:07.837771 31532 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:07.837973 30747 raft_consensus.cc:2393] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:07.838889 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.839746 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.840632 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.841446 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.842140 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:07.842635 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.843247 30596 raft_consensus.cc:3060] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:07.844420 31644 raft_consensus.cc:2804] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:07.844100 31612 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.844822 31644 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:07.844897 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.845296 31644 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.810532 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:07.846181 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:07.814043 31639 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:07.814324 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.813560 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:07.819739 30602 raft_consensus.cc:2393] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.819773 30645 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:07.817739 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.817338 31517 raft_consensus.cc:2804] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:07.822486 30752 raft_consensus.cc:2393] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:07.851468 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.852170 30752 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:07.853389 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:07.853735 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.854331 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.854920 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.856144 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.856441 31630 raft_consensus.cc:2749] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:07.857240 31517 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:07.857307 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.857911 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.858348 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.859103 30603 raft_consensus.cc:2393] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.822770 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.823869 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:07.859833 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.860404 30750 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 6: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:07.827827 30567 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.860848 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.794205 31567 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:07.829973 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:07.817772 31629 raft_consensus.cc:2804] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:04:07.836944 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.854938 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.862669 30600 raft_consensus.cc:2468] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:04:07.863272 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.863881 30598 raft_consensus.cc:2468] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:07.864691 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:07.865406 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 46840.6us
I20251025 20:04:07.862099 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.829815 31549 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:07.869769 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:07.870878 30750 raft_consensus.cc:2393] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:07.855015 30604 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:07.871897 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.857972 31639 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.872092 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.872947 31549 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.873572 30745 raft_consensus.cc:1240] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:07.874225 31639 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.857759 31517 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:07.875274 31549 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:07.868222 31567 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:07.876392 31567 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:07.868094 31629 raft_consensus.cc:493] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:07.876905 31629 raft_consensus.cc:3060] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:07.875741 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:07.880827 30745 raft_consensus.cc:1240] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:07.881119 30675 raft_consensus.cc:3060] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.881345 30567 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.882812 30750 raft_consensus.cc:1240] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:07.882999 30745 raft_consensus.cc:1240] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:07.883989 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:07.885777 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:07.887382 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:07.888195 30679 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:07.891505 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:07.891696 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.892758 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.894143 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.893795 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.895967 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.899801 30719 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:07.836220 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.851975 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:07.892149 30674 raft_consensus.cc:2468] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:07.904843 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.921937 30603 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:07.922890 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.904232 30602 raft_consensus.cc:2393] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.908890 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.904134 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.904601 31530 consensus_queue.cc:1059] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20251025 20:04:07.914254 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.914722 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:07.829519 31607 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.917045 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:07.923717 30603 raft_consensus.cc:2410] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:07.933905 30599 raft_consensus.cc:2393] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.944574 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:07.944808 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 88976.8us
I20251025 20:04:07.945205 30603 raft_consensus.cc:3060] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.943262 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.926059 30800 tablet_bootstrap.cc:654] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:07.946380 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.934433 30716 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:07.935866 31681 consensus_queue.cc:1059] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:04:07.936630 30671 raft_consensus.cc:2393] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.879829 31612 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:07.904134 31560 raft_consensus.cc:2804] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:07.940192 30719 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:07.940742 30600 raft_consensus.cc:3060] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:07.942058 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.942381 30675 raft_consensus.cc:2468] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:07.947098 30602 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 6: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:07.931413 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:04:07.977048 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:07.985739 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:07.987586 30602 raft_consensus.cc:3060] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:07.967967 30598 raft_consensus.cc:2468] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:07.985009 30745 raft_consensus.cc:1275] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251025 20:04:07.991356 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:07.945942 31499 raft_consensus.cc:2804] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:07.992515 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:07.993916 30745 raft_consensus.cc:2393] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:07.950685 31614 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:07.919420 31645 consensus_queue.cc:1048] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
W20251025 20:04:07.983732 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 102434us
W20251025 20:04:07.995537 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 55407.5us
I20251025 20:04:07.999559 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:08.000680 30753 raft_consensus.cc:2393] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:07.957173 31626 raft_consensus.cc:2804] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:08.006392 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:08.009194 30567 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:07.966269 31290 raft_consensus.cc:493] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:07.938860 31644 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:07.966924 31560 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:04:08.019855 30645 leader_election.cc:336] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.838s (SENT)
I20251025 20:04:08.020656 30602 raft_consensus.cc:2468] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:04:08.022677 30645 leader_election.cc:336] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.970s (SENT)
I20251025 20:04:08.023113 30645 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.023494 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.024529 30602 raft_consensus.cc:2393] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
W20251025 20:04:08.024724 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 158655us
I20251025 20:04:08.022990 31604 consensus_queue.cc:1048] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20251025 20:04:08.025857 30596 raft_consensus.cc:2468] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:08.017884 30744 raft_consensus.cc:2393] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:07.995625 31499 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:04:08.027884 30645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:08.014179 31626 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:07.998147 31641 raft_consensus.cc:2749] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:07.990530 31632 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.031102 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.038702 30596 raft_consensus.cc:2393] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:08.039667 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:08.040971 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.038707 31628 raft_consensus.cc:2804] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:08.042598 31628 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:08.043120 31628 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.032112 31499 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.044806 30596 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 6: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:08.023952 31290 raft_consensus.cc:515] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.045678 30601 raft_consensus.cc:2393] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:08.021625 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:08.046049 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.046260 30599 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:08.047305 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.046870 30596 raft_consensus.cc:2376] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2. Re-sending same reply.
I20251025 20:04:08.047881 30599 raft_consensus.cc:3060] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.023243 31560 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.026664 31630 raft_consensus.cc:2749] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:04:08.001046 30625 debug-util.cc:398] Leaking SignalData structure 0x7b0800199ac0 after lost signal to thread 29446
W20251025 20:04:08.059659 30625 debug-util.cc:398] Leaking SignalData structure 0x7b08001d5080 after lost signal to thread 30536
W20251025 20:04:08.061043 30625 debug-util.cc:398] Leaking SignalData structure 0x7b08002c94a0 after lost signal to thread 30628
I20251025 20:04:08.062085 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:08.062988 31607 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:08.053140 30753 raft_consensus.cc:1240] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:08.015779 31614 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.058395 31632 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.029734 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:08.028487 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.065582 30602 raft_consensus.cc:2393] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 1: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:08.034327 31626 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:08.067566 30751 raft_consensus.cc:1240] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.077154 30602 raft_consensus.cc:1240] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:08.085215 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 530) took 1990 ms. Trace:
I20251025 20:04:08.087692 30602 rpcz_store.cc:276] 1025 20:04:06.094360 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:08.076198 (+1981838us) service_pool.cc:225] Handling call
1025 20:04:08.076796 (+ 598us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:08.083408 (+ 6612us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:08.085142 (+ 1734us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:08.048671 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:08.096174 30719 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.077517 30603 raft_consensus.cc:2468] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:08.097728 30750 raft_consensus.cc:1275] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:08.037528 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:08.063427 31290 leader_election.cc:290] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.099037 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:08.098935 30719 leader_election.cc:304] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.101282 30603 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
W20251025 20:04:08.093544 30625 debug-util.cc:398] Leaking SignalData structure 0x7b08002a6b40 after lost signal to thread 30703
I20251025 20:04:08.092782 30595 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:08.110939 30625 debug-util.cc:398] Leaking SignalData structure 0x7b0800294420 after lost signal to thread 30704
W20251025 20:04:08.181114 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 171763us
I20251025 20:04:08.183197 30567 leader_election.cc:304] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:04:08.183748 30625 debug-util.cc:398] Leaking SignalData structure 0x7b080030d000 after lost signal to thread 30777
PC: @ 0x0 (unknown)
W20251025 20:04:08.109395 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 83536.7us
I20251025 20:04:08.188225 30570 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.142010 30752 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:08.117103 30645 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:08.115546 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 116772us
I20251025 20:04:08.133409 31517 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.199153 30716 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:08.201768 31517 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.111569 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.202195 30800 tablet_bootstrap.cc:492] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:04:08.203297 31517 raft_consensus.cc:2804] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:04:08.112778 30604 raft_consensus.cc:2468] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:08.203583 30716 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:04:08.197585 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 80338.3us
I20251025 20:04:08.204233 31517 raft_consensus.cc:697] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:08.216959 30800 ts_tablet_manager.cc:1403] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.498s user 0.030s sys 0.030s
I20251025 20:04:08.144807 31560 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.214188 30570 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.223740 31695 raft_consensus.cc:2749] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:08.224478 31560 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.140571 31567 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.104521 31607 raft_consensus.cc:2804] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:08.226634 30597 raft_consensus.cc:2393] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:08.226552 31607 raft_consensus.cc:697] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.227967 30570 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.228474 31567 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.228123 31607 consensus_queue.cc:237] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.104372 31632 consensus_queue.cc:1048] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20251025 20:04:08.103654 31641 raft_consensus.cc:2749] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:04:08.092303 31645 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:08.232343 30595 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:08.233024 31645 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.234210 31560 raft_consensus.cc:2749] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:08.236093 31645 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:08.236284 30570 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.103104 31690 raft_consensus.cc:2804] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:08.209328 31268 raft_consensus.cc:2804] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:08.205957 31697 raft_consensus.cc:2749] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:08.237388 31268 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:08.238032 31268 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.238178 31567 raft_consensus.cc:2749] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:08.238606 31690 raft_consensus.cc:697] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:08.240487 30570 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.240110 31690 consensus_queue.cc:237] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.241292 30595 raft_consensus.cc:3055] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:08.143458 31644 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.210355 31290 raft_consensus.cc:2804] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:08.242345 31567 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.242658 31290 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:08.243250 31290 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.243681 30598 raft_consensus.cc:2368] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2.
I20251025 20:04:08.212658 31694 raft_consensus.cc:2749] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:08.130049 31614 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:08.106127 30719 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.216801 31517 consensus_queue.cc:237] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.211280 31610 consensus_queue.cc:1048] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251025 20:04:08.247074 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 140896us
I20251025 20:04:08.247326 31610 raft_consensus.cc:2749] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:08.191632 30596 raft_consensus.cc:1275] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:04:08.135049 31629 raft_consensus.cc:515] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.248555 30719 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.217115 30716 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.092126 31532 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.251374 31629 leader_election.cc:290] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.251730 30716 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:08.252420 31694 raft_consensus.cc:2804] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:08.233696 30603 raft_consensus.cc:1240] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:08.253068 31694 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:08.252583 31532 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.254401 30601 raft_consensus.cc:2393] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 1: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:08.241709 30595 raft_consensus.cc:740] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:08.185367 30567 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.270123 31532 raft_consensus.cc:2749] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251025 20:04:08.271425 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 86494.3us
I20251025 20:04:08.255875 31614 raft_consensus.cc:2749] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:08.256880 31641 raft_consensus.cc:2804] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:08.259531 30719 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.252696 30596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 442) took 1052 ms. Trace:
W20251025 20:04:08.267313 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 50723.1us
I20251025 20:04:08.278465 30596 rpcz_store.cc:276] 1025 20:04:07.199786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:08.189325 (+989539us) service_pool.cc:225] Handling call
1025 20:04:08.190430 (+ 1105us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:08.252289 (+ 61859us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:08.252597 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:08.279878 30645 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34886 dropped due to backpressure: service queue is full with 50 items [suppressed 69 similar messages]
I20251025 20:04:08.281114 30596 raft_consensus.cc:1275] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 20:04:08.281440 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:08.282811 30645 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.140729 31626 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.253566 31694 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.272125 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:08.273670 31532 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:08.285431 30671 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:08.285557 31532 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.277824 31641 raft_consensus.cc:697] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.286964 31629 consensus_queue.cc:1059] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251025 20:04:08.287838 31641 consensus_queue.cc:237] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.288899 31644 raft_consensus.cc:3055] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:08.289134 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:08.289388 31644 raft_consensus.cc:740] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:08.290732 31696 raft_consensus.cc:2749] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:08.291517 30567 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.290773 31644 consensus_queue.cc:260] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.284199 31610 raft_consensus.cc:2804] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:08.142807 31499 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.275635 30595 consensus_queue.cc:260] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.306092 30800 raft_consensus.cc:359] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.307193 30800 raft_consensus.cc:385] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:08.307301 30595 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.307549 30800 raft_consensus.cc:740] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:04:08.308552 31499 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:08.308410 30800 consensus_queue.cc:260] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.309967 30746 raft_consensus.cc:1240] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.310706 30800 ts_tablet_manager.cc:1434] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.057s user 0.008s sys 0.001s
I20251025 20:04:08.310547 30567 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:08.284183 30719 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:08.318137 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:08.310560 31626 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:08.319151 30600 raft_consensus.cc:2468] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:08.319571 31628 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.320955 30600 raft_consensus.cc:1240] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.319928 31686 raft_consensus.cc:3055] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:08.323884 30597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 424) took 1126 ms. Trace:
I20251025 20:04:08.322572 31610 raft_consensus.cc:697] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:08.330312 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 46296.5us
I20251025 20:04:08.336985 30719 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.331636 30596 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:08.337368 31610 consensus_queue.cc:237] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.339931 31290 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.327415 31644 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.341985 31604 raft_consensus.cc:2804] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:08.342464 31604 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:08.342844 31604 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:08.342958 30719 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.343829 31491 raft_consensus.cc:2804] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:08.344236 31626 raft_consensus.cc:2804] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:08.326718 30597 rpcz_store.cc:276] 1025 20:04:07.195634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:07.734659 (+539025us) service_pool.cc:225] Handling call
1025 20:04:08.321726 (+587067us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b54001c20d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:08.322274 (+ 548us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":975707264}
I20251025 20:04:08.333920 31632 raft_consensus.cc:2749] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:04:08.346741 30716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:08.346905 30601 raft_consensus.cc:1240] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:08.347285 30597 raft_consensus.cc:1275] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:08.334388 30604 raft_consensus.cc:1240] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:08.335907 31532 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.323256 31686 raft_consensus.cc:740] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:08.351699 31626 raft_consensus.cc:697] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.324152 31690 raft_consensus.cc:2764] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251025 20:04:08.358197 30719 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.345731 31491 raft_consensus.cc:697] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:08.373847 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 92185.4us
W20251025 20:04:08.383559 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 100908us
I20251025 20:04:08.384989 30676 raft_consensus.cc:1240] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.390791 30599 raft_consensus.cc:2468] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:08.392405 30599 raft_consensus.cc:1240] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.393512 30598 raft_consensus.cc:1240] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.394402 31628 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.394418 30602 raft_consensus.cc:1240] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.395140 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.397614 31268 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.399295 31414 raft_consensus.cc:2804] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
W20251025 20:04:08.399405 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 53785.3us
I20251025 20:04:08.399997 31414 raft_consensus.cc:697] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.400130 31268 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:08.401114 31268 consensus_queue.cc:1059] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:04:08.401284 31414 consensus_queue.cc:237] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.402170 31290 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.403373 30597 raft_consensus.cc:1240] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.397782 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 437) took 1199 ms. Trace:
I20251025 20:04:08.403522 30570 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:08.410171 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 57849.9us
I20251025 20:04:08.410494 31686 consensus_queue.cc:260] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.411818 31686 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.411880 30570 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.412947 31690 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:08.413776 31690 raft_consensus.cc:2749] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:08.414409 30600 raft_consensus.cc:1240] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.415313 30570 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:08.415983 30600 raft_consensus.cc:1240] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.405027 30597 raft_consensus.cc:1240] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.409607 30598 raft_consensus.cc:1240] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.423525 30595 raft_consensus.cc:1275] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20251025 20:04:08.389544 31567 raft_consensus.cc:2749] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:08.409358 30602 rpcz_store.cc:276] 1025 20:04:07.198347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:08.092524 (+894177us) service_pool.cc:225] Handling call
1025 20:04:08.394045 (+301521us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:08.395964 (+ 1919us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:08.397433 (+ 1469us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b5400040950. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3944d 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:08.397658 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":692417408}
I20251025 20:04:08.424443 30595 pending_rounds.cc:85] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957: Aborting all ops after (but not including) 1
I20251025 20:04:08.368603 31694 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.416848 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:08.428447 30595 pending_rounds.cc:107] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251025 20:04:08.434322 30595 raft_consensus.cc:2889] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:04:08.432072 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.406782 31428 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:08.423525 31628 raft_consensus.cc:2764] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
*** SIGABRT (@0x3e800007303) received by PID 29443 (TID 0x7fd81a680840) from PID 29443; stack trace: ***
@ 0x2cd620 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2cfb03 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20251025 20:04:08.443410 30604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 448) took 1241 ms. Trace:
I20251025 20:04:08.443879 30677 raft_consensus.cc:1240] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20251025 20:04:08.417727 31626 consensus_queue.cc:237] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.438198 31428 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.437356 31549 consensus_queue.cc:1059] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20251025 20:04:08.451004 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:08.460675 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34886 (request call id 440) took 1261 ms. Trace:
I20251025 20:04:08.454238 31698 raft_consensus.cc:3055] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:08.463836 30597 raft_consensus.cc:1240] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.461090 30603 rpcz_store.cc:276] 1025 20:04:07.199246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:08.107449 (+908203us) service_pool.cc:225] Handling call
1025 20:04:08.233209 (+125760us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:08.253172 (+ 19963us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:08.458152 (+204980us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b5400308ed0. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3944d 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:08.460608 (+ 2456us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":288054144}
I20251025 20:04:08.466641 30671 raft_consensus.cc:2393] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 4: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:08.467480 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.468324 30670 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.463836 31698 raft_consensus.cc:740] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.471122 31698 consensus_queue.cc:260] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.472965 31698 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.459378 30599 raft_consensus.cc:1240] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:08.483963 30677 raft_consensus.cc:1275] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:08.443977 30604 rpcz_store.cc:276] 1025 20:04:07.201459 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:08.205195 (+1003736us) service_pool.cc:225] Handling call
1025 20:04:08.325968 (+120773us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:08.351610 (+ 25642us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:08.443067 (+ 91457us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54001c25d0. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3944d 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:08.443303 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":268221824}
I20251025 20:04:08.486614 30719 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.489370 30600 raft_consensus.cc:1240] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
W20251025 20:04:08.489602 31679 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20251025 20:04:08.479317 31607 consensus_queue.cc:1048] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
I20251025 20:04:08.495517 31610 raft_consensus.cc:3055] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:08.496062 31610 raft_consensus.cc:740] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.457510 31604 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:08.502043 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 71942.1us
I20251025 20:04:08.504992 31491 consensus_queue.cc:237] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.501516 30672 raft_consensus.cc:1240] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.497225 31610 consensus_queue.cc:260] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.493685 31428 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:08.515539 30645 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.541497 30719 leader_election.cc:400] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2.
I20251025 20:04:08.542145 30719 leader_election.cc:403] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:04:08.543273 30570 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.546541 30676 raft_consensus.cc:1275] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251025 20:04:08.552065 31704 log.cc:927] Time spent T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.083s user 0.002s sys 0.000s
W20251025 20:04:08.554993 31703 log.cc:927] Time spent T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.183s user 0.000s sys 0.000s
W20251025 20:04:08.565781 31688 log.cc:927] Time spent T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.430s user 0.000s sys 0.005s
I20251025 20:04:08.562637 31290 raft_consensus.cc:2804] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:08.567580 31290 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:08.568041 31290 raft_consensus.cc:3060] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.564359 31679 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:08.570407 31679 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.572939 31679 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.579053 31607 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.583009 30678 raft_consensus.cc:1275] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20251025 20:04:08.587779 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 64303.6us
I20251025 20:04:08.557739 31549 raft_consensus.cc:2804] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:08.563390 31644 raft_consensus.cc:2749] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:08.563233 31694 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:08.608058 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 94712.9us
W20251025 20:04:08.609004 31706 log.cc:927] Time spent T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
W20251025 20:04:08.618999 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 143090us
I20251025 20:04:08.622871 31414 consensus_queue.cc:1059] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:08.623888 30670 raft_consensus.cc:2468] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:08.624085 31414 raft_consensus.cc:3055] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:08.624517 31414 raft_consensus.cc:740] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.611413 31549 raft_consensus.cc:697] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.634298 31604 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:08.635152 30645 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.636200 31696 raft_consensus.cc:2749] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251025 20:04:08.640563 31705 log.cc:927] Time spent T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20251025 20:04:08.644135 31610 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:08.643808 31549 consensus_queue.cc:237] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.649858 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:08.651399 31414 consensus_queue.cc:260] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.652619 31414 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.654107 31694 consensus_queue.cc:1059] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:04:08.655593 31694 raft_consensus.cc:3055] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:08.656075 31694 raft_consensus.cc:740] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.637845 31290 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.658469 31645 consensus_queue.cc:1048] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:04:08.658527 31268 consensus_queue.cc:1059] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:08.659452 31428 consensus_queue.cc:1059] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:04:08.659989 31423 consensus_queue.cc:1048] T 3f1269e0f1364a69b9a36ce877b0da5e P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:04:08.660820 31624 raft_consensus.cc:3055] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:08.660995 31290 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:08.661293 31624 raft_consensus.cc:740] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.658906 31694 consensus_queue.cc:260] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.662086 31428 consensus_queue.cc:1059] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:04:08.662451 31624 consensus_queue.cc:260] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.663296 31645 consensus_queue.cc:1059] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:04:08.663587 31423 consensus_queue.cc:1059] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:04:08.658944 31604 consensus_queue.cc:1059] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20251025 20:04:08.660189 31268 raft_consensus.cc:3055] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:08.665446 31499 raft_consensus.cc:3055] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:08.674540 31499 raft_consensus.cc:740] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.675304 31499 consensus_queue.cc:260] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.676327 31499 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.649698 31626 consensus_queue.cc:1059] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20251025 20:04:08.666680 31423 raft_consensus.cc:3055] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:08.678586 31423 raft_consensus.cc:740] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.679140 31626 raft_consensus.cc:3055] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:08.679651 31423 consensus_queue.cc:260] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.681366 31423 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.667733 31694 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.665983 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:08.686304 30599 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
W20251025 20:04:08.664371 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 49874.3us
I20251025 20:04:08.666790 31645 raft_consensus.cc:3055] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:08.688130 31645 raft_consensus.cc:740] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.688443 30570 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.689561 31679 raft_consensus.cc:2749] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:08.689059 31645 consensus_queue.cc:260] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.691009 31645 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.667439 31428 raft_consensus.cc:3055] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:08.679651 31626 raft_consensus.cc:740] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.693418 31428 raft_consensus.cc:740] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.694221 31626 consensus_queue.cc:260] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.694586 31428 consensus_queue.cc:260] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.695907 31428 raft_consensus.cc:3060] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.696161 31626 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.667547 31268 raft_consensus.cc:740] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.666038 31624 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.699144 30570 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.700613 30570 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.701676 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.702536 31268 consensus_queue.cc:260] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.710793 31268 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.714706 30601 raft_consensus.cc:1240] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.707037 31607 raft_consensus.cc:2749] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2.
I20251025 20:04:08.707087 30677 raft_consensus.cc:1240] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.708951 30673 raft_consensus.cc:1275] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:08.719985 31689 raft_consensus.cc:2749] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:08.706051 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:08.689811 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:08.725391 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.735739 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.736853 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.737151 30595 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:08.737620 30676 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:04:08.737178 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 53724.3us
I20251025 20:04:08.720806 31679 raft_consensus.cc:2749] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:08.740883 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:08.742545 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.734851 30599 raft_consensus.cc:1275] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:08.742538 30673 raft_consensus.cc:1240] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.743973 30597 raft_consensus.cc:3055] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:08.744382 30597 raft_consensus.cc:740] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:08.746389 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.746750 30600 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.747032 30716 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:08.749282 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.739516 30670 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
W20251025 20:04:08.754719 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 54212.8us
I20251025 20:04:08.757469 31485 consensus_queue.cc:1048] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20251025 20:04:08.758160 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.760113 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.734418 31641 consensus_queue.cc:1048] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251025 20:04:08.762280 30672 raft_consensus.cc:1240] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.763854 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.765443 30675 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.771368 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.771344 31689 consensus_queue.cc:1059] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20251025 20:04:08.773981 31478 consensus_queue.cc:1048] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20251025 20:04:08.757694 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.754379 31694 raft_consensus.cc:2804] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:08.774523 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.779652 30678 raft_consensus.cc:1240] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.778862 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.777055 30679 raft_consensus.cc:1240] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.771710 30604 raft_consensus.cc:3055] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:08.794445 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.786439 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:08.796190 30674 raft_consensus.cc:1240] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.785241 30597 consensus_queue.cc:260] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.813831 30673 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.799109 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:08.804025 30569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:08.807824 30595 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:08.806556 30604 raft_consensus.cc:740] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
W20251025 20:04:08.821084 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:08.820474 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.820931 30604 consensus_queue.cc:260] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.821798 30601 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:04:08.822345 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 44609.5us
I20251025 20:04:08.827250 30570 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.828586 30600 raft_consensus.cc:2468] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:08.829209 30570 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.822686 30597 raft_consensus.cc:3060] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:08.823583 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:08.830600 30602 raft_consensus.cc:3060] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:08.830638 30604 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:08.835116 30601 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:08.835906 31644 raft_consensus.cc:2749] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251025 20:04:08.836630 31644 raft_consensus.cc:2749] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:08.836529 30600 raft_consensus.cc:1240] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:08.786686 31686 consensus_queue.cc:1059] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20251025 20:04:08.840711 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:08.841609 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 72412.7us
I20251025 20:04:08.848011 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.848960 30676 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:08.857456 30570 leader_election.cc:304] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.859724 30673 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:08.818081 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.882536 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:08.859189 31694 raft_consensus.cc:697] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.865553 30670 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
W20251025 20:04:08.870009 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:08.871034 30719 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.898810 30675 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:08.874172 30678 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:08.874676 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.891073 30597 raft_consensus.cc:1275] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251025 20:04:08.906107 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 49079.7us
I20251025 20:04:08.908396 31628 raft_consensus.cc:2749] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:04:08.909133 30570 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:08.905977 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:08.910605 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:08.924592 30597 raft_consensus.cc:1275] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:08.924713 30602 raft_consensus.cc:2468] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:08.909929 31628 consensus_queue.cc:1059] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.018s
I20251025 20:04:08.928604 31628 raft_consensus.cc:3055] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:08.928973 31628 raft_consensus.cc:740] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
@ 0x7fd821f44980 (unknown) at ??:0
I20251025 20:04:08.936105 30600 raft_consensus.cc:1240] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
W20251025 20:04:08.936713 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 65803.2us
I20251025 20:04:08.933725 30570 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:08.940280 30670 raft_consensus.cc:1240] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:08.944455 30604 raft_consensus.cc:1275] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251025 20:04:08.945237 31679 raft_consensus.cc:3055] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:08.935508 30671 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.929975 31628 consensus_queue.cc:260] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:08.927188 31567 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:08.971789 30719 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.922976 31641 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:08.944936 30604 pending_rounds.cc:85] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957: Aborting all ops after (but not including) 2
I20251025 20:04:08.910058 31696 consensus_queue.cc:1059] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
W20251025 20:04:08.981541 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 93952.8us
I20251025 20:04:08.945773 31679 raft_consensus.cc:740] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:08.980418 30604 pending_rounds.cc:107] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251025 20:04:08.976440 31641 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:08.981125 31690 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:08.994910 30719 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.997386 30719 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.988611 31485 raft_consensus.cc:2749] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:08.999208 30674 raft_consensus.cc:1240] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:09.000240 30567 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:09.001397 30673 raft_consensus.cc:1275] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:09.004554 30719 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:08.990907 31614 consensus_queue.cc:1048] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20251025 20:04:09.010180 31478 raft_consensus.cc:2804] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:09.010823 31478 raft_consensus.cc:697] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:09.011891 30677 raft_consensus.cc:3055] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:09.011831 31478 consensus_queue.cc:237] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.012806 30671 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:09.014349 31485 raft_consensus.cc:2804] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:09.014953 31485 raft_consensus.cc:697] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:08.978899 31414 consensus_queue.cc:1059] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20251025 20:04:09.003352 31694 consensus_queue.cc:237] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.015272 30719 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:09.012357 30677 raft_consensus.cc:740] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:09.018658 31485 consensus_queue.cc:237] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.021648 30716 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:08.990720 30604 raft_consensus.cc:2889] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:04:09.023602 31610 consensus_queue.cc:1048] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:09.023932 31290 consensus_queue.cc:1059] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:04:09.024446 31614 raft_consensus.cc:2804] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:09.024995 31607 raft_consensus.cc:2804] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:09.016798 31644 raft_consensus.cc:2749] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251025 20:04:09.025686 31696 consensus_queue.cc:1048] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:04:09.026080 31698 consensus_queue.cc:1048] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:04:09.026537 30670 raft_consensus.cc:1275] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:08.995786 31641 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.996080 31628 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:08.999612 31549 raft_consensus.cc:2749] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251025 20:04:09.045797 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 45660.6us
I20251025 20:04:09.064533 30567 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:09.065851 30678 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:09.055851 31614 raft_consensus.cc:697] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:09.090334 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.062624 31690 raft_consensus.cc:697] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:09.045042 31727 consensus_queue.cc:1059] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20251025 20:04:09.057200 30677 consensus_queue.cc:260] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.053992 31499 consensus_queue.cc:1048] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251025 20:04:09.059588 31607 raft_consensus.cc:697] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:09.071256 31689 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20251025 20:04:09.068186 31645 consensus_queue.cc:1048] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
W20251025 20:04:09.124231 31704 log.cc:927] Time spent T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20251025 20:04:09.114500 31268 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:09.137216 30600 raft_consensus.cc:1275] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:04:09.120267 31732 raft_consensus.cc:2749] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:09.145326 30677 raft_consensus.cc:3060] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:09.120729 31626 consensus_queue.cc:1059] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20251025 20:04:09.067360 31679 consensus_queue.cc:260] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.122304 31414 consensus_queue.cc:1048] T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20251025 20:04:09.159158 30603 raft_consensus.cc:3055] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:09.162565 30603 raft_consensus.cc:740] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:09.158818 30674 raft_consensus.cc:3055] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Stepping down as leader of term 2
W20251025 20:04:09.173708 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 109214us
I20251025 20:04:09.175283 30567 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:09.168705 30674 raft_consensus.cc:740] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:09.151503 31604 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:09.137133 31491 consensus_queue.cc:1048] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.029s
I20251025 20:04:09.186558 30603 consensus_queue.cc:260] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.192260 30603 raft_consensus.cc:3060] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:09.159126 31268 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.187033 31679 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:09.188225 30674 consensus_queue.cc:260] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.198928 30679 raft_consensus.cc:3055] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
W20251025 20:04:09.193538 31726 log.cc:927] Time spent T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20251025 20:04:09.204665 31731 raft_consensus.cc:3055] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:09.205098 31731 raft_consensus.cc:740] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:09.206049 31731 consensus_queue.cc:260] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.207580 31731 raft_consensus.cc:3060] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:09.191821 31698 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:09.210314 31698 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.211582 31624 raft_consensus.cc:3055] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:09.212050 31624 raft_consensus.cc:740] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:09.212414 31698 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:09.202180 30679 raft_consensus.cc:740] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
W20251025 20:04:09.195995 31729 log.cc:927] Time spent T 4614bdb7f750431fa840cabccf782b4b P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
W20251025 20:04:09.220870 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 45648.9us
W20251025 20:04:09.218276 30569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.187378 31604 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.230597 30600 raft_consensus.cc:1240] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:09.231488 30595 raft_consensus.cc:1275] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:04:09.234274 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:09.234172 31641 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.235270 30595 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 6.
I20251025 20:04:09.177130 31614 consensus_queue.cc:237] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.192068 31737 raft_consensus.cc:2749] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:09.218931 30599 raft_consensus.cc:1275] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:04:09.244030 30604 raft_consensus.cc:1275] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:09.188834 31690 consensus_queue.cc:237] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.249540 31567 raft_consensus.cc:2749] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:09.212869 31624 consensus_queue.cc:260] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.272374 30674 raft_consensus.cc:3060] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:09.274024 30567 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:09.272374 30603 raft_consensus.cc:1275] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:09.272388 30679 consensus_queue.cc:260] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.191040 31607 consensus_queue.cc:237] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.289847 30677 raft_consensus.cc:1275] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:09.244088 31268 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:09.198014 31738 consensus_queue.cc:1059] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.091s
W20251025 20:04:09.293620 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 186706us
I20251025 20:04:09.282872 31641 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:09.242112 31698 consensus_queue.cc:1048] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20251025 20:04:09.296844 30670 raft_consensus.cc:3055] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:09.308513 30679 raft_consensus.cc:3060] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:09.317606 30719 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:09.297953 31604 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:09.305464 31567 raft_consensus.cc:2749] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:09.324090 31428 consensus_queue.cc:1048] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20251025 20:04:09.335608 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.338542 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.305444 31624 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:09.312665 30670 raft_consensus.cc:740] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
W20251025 20:04:09.372126 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 54644us
I20251025 20:04:09.299515 31732 consensus_queue.cc:1059] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20251025 20:04:09.396922 30752 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
W20251025 20:04:09.410625 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 249575us
I20251025 20:04:09.390547 31428 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251025 20:04:09.412693 30675 raft_consensus.cc:3055] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:09.381589 31732 raft_consensus.cc:3055] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:09.416224 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:09.414448 30675 raft_consensus.cc:740] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:09.422052 31732 raft_consensus.cc:740] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
W20251025 20:04:09.424508 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.426523 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:09.427170 30645 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:09.429989 30603 raft_consensus.cc:1275] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:09.431510 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:09.502863 30595 raft_consensus.cc:2468] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:09.434069 31485 consensus_queue.cc:1048] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:04:09.424551 31753 consensus_queue.cc:1048] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20251025 20:04:09.439193 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:09.506666 30600 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:09.442525 30749 raft_consensus.cc:2468] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:09.442960 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:09.443217 31739 raft_consensus.cc:3055] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:09.451622 30670 consensus_queue.cc:260] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:09.455040 31745 log.cc:927] Time spent T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.120s user 0.000s sys 0.002s
I20251025 20:04:09.461575 31721 raft_consensus.cc:3055] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:09.470860 30752 raft_consensus.cc:1240] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:09.471843 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.473716 30747 raft_consensus.cc:1240] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:09.524744 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:09.478763 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:09.410406 31614 consensus_queue.cc:1048] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20251025 20:04:09.414422 31428 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:09.485582 30501 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39172 (request call id 90) took 6737 ms. Trace:
I20251025 20:04:09.398958 31567 consensus_queue.cc:1059] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
W20251025 20:04:09.486491 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 62825.6us
I20251025 20:04:09.545650 30567 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:09.380306 31744 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:09.492888 30679 raft_consensus.cc:1275] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:09.408154 31698 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:09.507305 30674 raft_consensus.cc:1275] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:04:09.509119 30500 catalog_manager.cc:5649] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
W20251025 20:04:09.513531 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 106654us
W20251025 20:04:09.517194 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.522564 30603 raft_consensus.cc:1275] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 20:04:09.551638 30645 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:09.552420 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.378733 31749 consensus_queue.cc:1059] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.103s
I20251025 20:04:09.511771 30599 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:09.519430 31721 raft_consensus.cc:740] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:09.558581 30600 raft_consensus.cc:1240] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
W20251025 20:04:09.560513 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.510870 31739 raft_consensus.cc:740] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
W20251025 20:04:09.548281 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.543383 31428 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Advancing to term 7
I20251025 20:04:09.543782 31268 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:09.533123 31689 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:09.584244 30500 catalog_manager.cc:5649] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
W20251025 20:04:09.588083 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 63778.2us
W20251025 20:04:09.586705 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.561717 31738 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election won for term 4
W20251025 20:04:09.620358 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 74808.2us
I20251025 20:04:09.623288 30567 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:04:09.624439 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20251025 20:04:09.624960 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:09.626623 31718 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20251025 20:04:09.561553 30675 consensus_queue.cc:260] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.633827 31733 raft_consensus.cc:2804] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:09.634532 31733 raft_consensus.cc:697] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:09.634874 31751 raft_consensus.cc:2804] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:09.635773 31733 consensus_queue.cc:237] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.564122 31698 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.639160 30500 catalog_manager.cc:5649] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 1 to 3, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 3 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:09.640044 31698 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:09.640360 30670 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:09.641045 30500 catalog_manager.cc:5649] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:09.643467 30678 raft_consensus.cc:1275] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20251025 20:04:09.643852 30678 pending_rounds.cc:85] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 0
I20251025 20:04:09.644106 30678 pending_rounds.cc:107] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20251025 20:04:09.644356 30678 raft_consensus.cc:2889] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251025 20:04:09.644467 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 85049.8us
W20251025 20:04:09.631378 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
@ 0x7fd81b340fb7 gsignal at ??:0
I20251025 20:04:09.652889 31739 consensus_queue.cc:260] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.654233 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:09.591974 31290 consensus_queue.cc:1048] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20251025 20:04:09.653682 30500 catalog_manager.cc:5649] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 1 to 3, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 3 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:09.622997 31686 raft_consensus.cc:2804] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:09.624366 31721 consensus_queue.cc:260] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.539608 30501 rpcz_store.cc:276] 1025 20:04:02.842475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:02.845262 (+ 2787us) service_pool.cc:225] Handling call
1025 20:04:09.474062 (+6628800us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 20:04:08.060669 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 20:04:08.061314 (+ 645us) write_op.cc:432] Acquiring schema lock in shared mode
1025 20:04:08.061349 (+ 35us) write_op.cc:435] Acquired schema lock
1025 20:04:08.061370 (+ 21us) tablet.cc:660] Decoding operations
1025 20:04:08.062102 (+ 732us) write_op.cc:620] Acquiring the partition lock for write op
1025 20:04:08.062188 (+ 86us) write_op.cc:641] Partition lock acquired for write op
1025 20:04:08.062220 (+ 32us) tablet.cc:683] Acquiring locks for 4 operations
1025 20:04:08.062629 (+ 409us) tablet.cc:699] Row locks acquired
1025 20:04:08.062649 (+ 20us) write_op.cc:260] PREPARE: finished
1025 20:04:08.062831 (+ 182us) write_op.cc:270] Start()
1025 20:04:08.062945 (+ 114us) write_op.cc:276] Timestamp: P: 1761422648062789 usec, L: 0
1025 20:04:08.062966 (+ 21us) op_driver.cc:348] REPLICATION: starting
1025 20:04:08.112442 (+ 49476us) log.cc:844] Serialized 1717 byte log entry
1025 20:04:08.316429 (+203987us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd81ea7059c 00007fd827849219 00007fd81ea8cc4b 00007fd81ea8aeef 00007fd81ea8d1b2 00007fd8216d42bb
1025 20:04:08.839249 (+522820us) op_driver.cc:464] REPLICATION: finished
1025 20:04:08.922026 (+ 82777us) write_op.cc:301] APPLY: starting
1025 20:04:08.924373 (+ 2347us) tablet.cc:1365] starting BulkCheckPresence
1025 20:04:08.980309 (+ 55936us) tablet.cc:1368] finished BulkCheckPresence
1025 20:04:08.987015 (+ 6706us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 20:04:09.181817 (+194802us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 20:04:09.251285 (+ 69468us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
1025 20:04:09.282298 (+ 31013us) write_op.cc:312] APPLY: finished
1025 20:04:09.327817 (+ 45519us) log.cc:844] Serialized 40 byte log entry
1025 20:04:09.328049 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:04:09.328205 (+ 156us) write_op.cc:454] Released schema lock
1025 20:04:09.328271 (+ 66us) write_op.cc:341] FINISH: Updating metrics
1025 20:04:09.328731 (+ 460us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:04:09.328770 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":95001,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":12806,"apply.run_cpu_time_us":5243,"apply.run_wall_time_us":476262,"num_ops":4,"prepare.queue_time_us":208385,"prepare.run_cpu_time_us":22100,"prepare.run_wall_time_us":504262,"raft.queue_time_us":127373,"raft.run_cpu_time_us":4651,"raft.run_wall_time_us":66155,"replication_time_us":753171,"spinlock_wait_cycles":34382080,"thread_start_us":619657,"threads_started":4,"wal-append.queue_time_us":469742}]]}
I20251025 20:04:09.644986 31268 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.662818 31742 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:09.677002 31739 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:09.677565 31742 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:09.663518 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 73258.8us
W20251025 20:04:09.680366 31762 log.cc:927] Time spent T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20251025 20:04:09.667104 30719 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:09.694883 30567 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:09.680323 31742 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:09.698513 31760 log.cc:927] Time spent T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20251025 20:04:09.556685 31744 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.696569 31738 raft_consensus.cc:697] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:09.701946 31744 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:09.644565 31751 raft_consensus.cc:697] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:09.681486 30746 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:09.740343 30567 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:09.596410 31689 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.672466 31478 consensus_queue.cc:1048] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20251025 20:04:09.732558 31721 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:09.754108 31567 consensus_queue.cc:1048] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:09.705611 31686 raft_consensus.cc:697] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:09.741231 30747 raft_consensus.cc:1240] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:09.693077 31749 consensus_queue.cc:1059] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20251025 20:04:09.739068 31679 raft_consensus.cc:2804] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
W20251025 20:04:09.756599 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 52253.8us
I20251025 20:04:09.587661 31732 consensus_queue.cc:260] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.786422 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.795083 31727 raft_consensus.cc:3055] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:09.795598 31727 raft_consensus.cc:740] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:09.760252 31690 raft_consensus.cc:2804] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:09.790630 31738 consensus_queue.cc:237] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.808583 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:09.809662 30604 raft_consensus.cc:2393] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
W20251025 20:04:09.809878 31756 log.cc:927] Time spent T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20251025 20:04:09.810909 30604 raft_consensus.cc:3055] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:09.811319 30604 raft_consensus.cc:740] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:09.792419 31686 consensus_queue.cc:237] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.808634 30595 raft_consensus.cc:1275] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:09.791390 31679 raft_consensus.cc:697] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:09.796557 31727 consensus_queue.cc:260] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.747974 31428 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:09.806440 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.808768 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:09.785661 30500 catalog_manager.cc:5649] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 0 to 1, leader changed from <none> to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 1 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:09.811719 31732 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:09.812129 30604 consensus_queue.cc:260] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:09.822250 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 141392us
I20251025 20:04:09.838313 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:09.843050 30670 raft_consensus.cc:1275] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251025 20:04:09.843802 31696 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20251025 20:04:09.843943 31733 consensus_queue.cc:1059] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:04:09.844836 31739 raft_consensus.cc:3055] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:09.845242 31739 raft_consensus.cc:740] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:09.844954 30500 catalog_manager.cc:5649] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 3 to 4, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:09.846242 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.809656 31751 consensus_queue.cc:237] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.847801 30745 raft_consensus.cc:1240] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 7 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 6. Current term is 7. Ops: []
I20251025 20:04:09.849027 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:09.847687 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.850241 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.815402 31644 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:09.847258 30747 raft_consensus.cc:2393] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:09.849606 30745 raft_consensus.cc:2468] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:09.770473 31754 consensus_queue.cc:1048] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.066s
I20251025 20:04:09.870815 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.872164 30752 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
W20251025 20:04:09.873245 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 206235us
I20251025 20:04:09.874351 30719 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:09.880636 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.789182 31718 consensus_queue.cc:1048] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20251025 20:04:09.895191 30746 raft_consensus.cc:2468] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:09.895612 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:09.896611 30746 raft_consensus.cc:1240] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
W20251025 20:04:09.889679 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.898756 31414 consensus_queue.cc:1048] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:04:09.899000 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:09.900065 31679 consensus_queue.cc:237] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.900907 31689 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:09.895387 30745 raft_consensus.cc:1275] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
@ 0x7fd81b342921 abort at ??:0
I20251025 20:04:09.906796 30752 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:09.908119 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.908888 30752 raft_consensus.cc:2393] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
W20251025 20:04:09.888530 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 120745us
I20251025 20:04:09.910885 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:09.911579 30676 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:04:09.913614 30570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:09.914192 30570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.914822 30570 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
@ 0x2d3616 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20251025 20:04:09.892935 31624 raft_consensus.cc:2804] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:09.919134 30677 raft_consensus.cc:3060] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:09.890878 31614 raft_consensus.cc:2804] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:09.846202 31739 consensus_queue.cc:260] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.921301 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:09.925566 30747 raft_consensus.cc:1275] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:09.927007 30747 raft_consensus.cc:1240] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:09.897517 31743 consensus_queue.cc:1048] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20251025 20:04:09.929204 31727 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:09.922955 31698 consensus_queue.cc:1048] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20251025 20:04:09.930652 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:09.904382 31268 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:09.932718 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.933593 30745 raft_consensus.cc:2393] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:09.907619 31499 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:09.934492 31499 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.934913 30745 raft_consensus.cc:1275] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:09.936600 31499 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:09.920570 31624 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:09.937695 31428 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:09.938485 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:09.915814 31644 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.936472 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:09.937773 31624 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:09.940439 31644 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:09.903199 31645 consensus_queue.cc:1048] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20251025 20:04:09.903390 31610 consensus_queue.cc:1048] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
W20251025 20:04:09.917135 30645 connection.cc:232] RPC call timeout handler was delayed by -1.8700086219998866s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.39540000000000003s for a response
W20251025 20:04:09.948546 30645 connection.cc:232] RPC call timeout handler was delayed by -1.7438194690000728s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1932s for a response
I20251025 20:04:09.957118 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:09.920598 31690 raft_consensus.cc:697] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
W20251025 20:04:09.962118 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 47422.7us
I20251025 20:04:09.963204 30570 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:09.924010 31614 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:09.976356 30677 raft_consensus.cc:1275] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:09.976984 30716 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:09.977409 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:09.978112 30677 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:09.953052 30599 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:09.960996 31690 consensus_queue.cc:237] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:09.986712 30677 raft_consensus.cc:1275] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251025 20:04:09.971451 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:09.990787 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 87932.3us
I20251025 20:04:09.975358 31614 raft_consensus.cc:3060] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:10.000790 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:10.001832 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:10.001866 30570 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:09.988493 31549 raft_consensus.cc:2804] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:10.010476 30716 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:09.994174 31739 raft_consensus.cc:3060] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:10.017513 30570 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:10.008684 31641 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.006942 31644 consensus_queue.cc:1059] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251025 20:04:10.024331 31751 raft_consensus.cc:2749] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251025 20:04:10.024468 31757 log.cc:927] Time spent T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.394s user 0.003s sys 0.000s
I20251025 20:04:10.025358 31772 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:10.010177 31549 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:10.042049 31549 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:10.034292 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:10.045188 30744 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:10.018576 31610 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.032362 31753 consensus_queue.cc:1048] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20251025 20:04:10.036041 30570 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
W20251025 20:04:10.081019 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 45075.2us
I20251025 20:04:10.082424 30749 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:10.082459 30570 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:10.082669 31751 raft_consensus.cc:2804] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election won for term 2
W20251025 20:04:10.084005 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 52446.9us
I20251025 20:04:10.085027 30719 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:10.032181 31721 raft_consensus.cc:2804] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251025 20:04:10.040292 31758 log.cc:927] Time spent T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.410s user 0.000s sys 0.003s
I20251025 20:04:10.033867 31644 consensus_queue.cc:1243] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer 783f8fd6d80141b7912f66a954a6f92e log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251025 20:04:10.012320 31742 consensus_queue.cc:1048] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20251025 20:04:10.104516 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.105044 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.106540 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.037876 31771 raft_consensus.cc:2804] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:10.114063 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:10.118656 31775 log.cc:927] Time spent T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20251025 20:04:10.038192 31769 raft_consensus.cc:2804] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:10.122592 30673 raft_consensus.cc:3055] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:10.096566 31721 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:10.123191 30673 raft_consensus.cc:740] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:10.115121 30597 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:10.032588 31641 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:10.132512 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 171918us
I20251025 20:04:10.123412 31721 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:10.134645 30675 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:10.102027 31694 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:10.102547 31772 raft_consensus.cc:697] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:10.114737 31751 raft_consensus.cc:697] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:10.087267 31610 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.136497 30673 consensus_queue.cc:260] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.123857 31689 raft_consensus.cc:2804] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:10.126621 31769 raft_consensus.cc:697] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
W20251025 20:04:10.250998 30645 connection.cc:232] RPC call timeout handler was delayed by -1.6679906429999392s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
I20251025 20:04:10.259456 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:10.261492 30675 raft_consensus.cc:1275] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251025 20:04:10.246518 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:10.241585 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:10.174309 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.162322 30600 raft_consensus.cc:1240] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:10.148679 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.263753 30673 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:04:10.255801 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.261008 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.136150 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.208772 31771 raft_consensus.cc:697] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:10.182438 31679 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.282106 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.282958 30599 raft_consensus.cc:2393] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
W20251025 20:04:10.286978 31778 log.cc:927] Time spent T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.154s user 0.000s sys 0.001s
I20251025 20:04:10.132028 31624 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.266840 31694 raft_consensus.cc:697] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:10.318521 30600 raft_consensus.cc:1240] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:10.269095 31689 raft_consensus.cc:697] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:10.321036 31686 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:10.321681 31628 raft_consensus.cc:3055] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:10.322268 31628 raft_consensus.cc:740] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:10.321959 31686 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:10.321753 31782 log.cc:927] Time spent T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
W20251025 20:04:10.291941 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.327527 30600 raft_consensus.cc:1240] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:10.294076 30670 raft_consensus.cc:1275] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251025 20:04:10.335139 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 248375us
I20251025 20:04:10.343577 30748 raft_consensus.cc:2393] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 4: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:10.351966 30675 raft_consensus.cc:1240] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:10.354367 30600 raft_consensus.cc:1240] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:10.357717 31610 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:10.308811 31679 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.366523 30598 raft_consensus.cc:1240] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:10.362008 31751 consensus_queue.cc:237] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.380656 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.443703 30600 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:04:10.379060 30645 connection.cc:232] RPC call timeout handler was delayed by -1.6490906429999086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1589s for a response
I20251025 20:04:10.396030 30673 raft_consensus.cc:1275] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251025 20:04:10.402606 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:10.402774 31785 log.cc:927] Time spent T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20251025 20:04:10.398523 31769 consensus_queue.cc:237] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.403682 31689 consensus_queue.cc:237] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:10.407831 31790 log.cc:927] Time spent T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20251025 20:04:10.419353 31686 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:10.421314 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:10.323172 31628 consensus_queue.cc:260] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:10.423218 31755 log.cc:927] Time spent T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.789s user 0.003s sys 0.003s
W20251025 20:04:10.424295 31784 log.cc:927] Time spent T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20251025 20:04:10.469959 30750 raft_consensus.cc:2410] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:10.424665 31772 consensus_queue.cc:237] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.427418 30751 raft_consensus.cc:1217] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Deduplicated request from leader. Original: 3.2->[4.3-4.3] Dedup: 4.3->[]
I20251025 20:04:10.429854 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.432121 30716 leader_election.cc:304] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:10.475972 30602 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
W20251025 20:04:10.476953 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 44972.6us
I20251025 20:04:10.432883 31641 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:10.433683 31694 consensus_queue.cc:237] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:10.341228 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.486579 31485 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.323503 31549 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:10.448288 30645 connection.cc:232] RPC call timeout handler was delayed by -1.641890642999897s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1597s for a response
W20251025 20:04:10.487452 30645 connection.cc:232] RPC call timeout handler was delayed by -1.6259266370000205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3183s for a response
I20251025 20:04:10.452952 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:10.453712 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.489352 30600 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:10.454461 30604 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:10.448196 31428 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:10.458760 31783 log.cc:927] Time spent T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20251025 20:04:10.459312 31771 consensus_queue.cc:237] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.463011 30674 raft_consensus.cc:1275] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:04:10.464401 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:10.495064 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:10.495543 30603 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Advancing to term 7
W20251025 20:04:10.487829 30645 connection.cc:232] RPC call timeout handler was delayed by -1.610390642999846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
W20251025 20:04:10.492447 30570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.480386 30751 raft_consensus.cc:1240] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251025 20:04:10.495003 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.493661 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.498517 30570 leader_election.cc:304] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:10.486598 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.498345 30598 raft_consensus.cc:2468] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:04:10.502293 30645 connection.cc:232] RPC call timeout handler was delayed by -1.6013906430000588s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1642s for a response
W20251025 20:04:10.502620 30645 connection.cc:232] RPC call timeout handler was delayed by -1.587890643000037s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
I20251025 20:04:10.481020 31686 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.490293 31478 raft_consensus.cc:2749] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:10.487172 31743 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.565258 30597 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:10.549278 30603 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 7.
I20251025 20:04:10.491531 31604 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.555202 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:10.520069 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:10.505085 31732 consensus_meta.cc:287] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.686s user 0.009s sys 0.013s
I20251025 20:04:10.582890 30598 raft_consensus.cc:1275] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:10.468371 31614 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.582901 30604 raft_consensus.cc:1275] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251025 20:04:10.589423 30602 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:10.588215 30604 pending_rounds.cc:85] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: Aborting all ops after (but not including) 2
I20251025 20:04:10.500691 31428 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.599571 30604 pending_rounds.cc:107] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251025 20:04:10.603099 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:10.603268 30604 raft_consensus.cc:2889] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251025 20:04:10.588151 30645 leader_election.cc:336] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 2.877s (SENT)
I20251025 20:04:10.606637 30645 leader_election.cc:304] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:10.577963 31744 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.556435 31290 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.503700 31485 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:10.597466 31727 consensus_meta.cc:287] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.668s user 0.008s sys 0.006s
I20251025 20:04:10.588941 31628 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:10.671528 30601 raft_consensus.cc:1240] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 6. Current term is 7. Ops: []
I20251025 20:04:10.674321 30598 raft_consensus.cc:1240] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
W20251025 20:04:10.677454 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.558483 31686 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.682351 30673 raft_consensus.cc:1275] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251025 20:04:10.618319 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.581393 31604 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.676082 30747 raft_consensus.cc:1240] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:10.740154 31604 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:10.611202 31624 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:10.612841 31739 raft_consensus.cc:2749] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:04:10.615904 31690 raft_consensus.cc:2749] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 20:04:10.616291 31718 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.741742 31718 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:10.689481 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.602284 31732 consensus_queue.cc:1048] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.081s
W20251025 20:04:10.688148 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:10.752396 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.752718 30752 raft_consensus.cc:1275] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:10.686501 31731 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.753314 30676 raft_consensus.cc:1275] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 20:04:10.753937 31731 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.684542 30671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 601) took 1557 ms. Trace:
I20251025 20:04:10.617992 31290 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.618494 31744 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.764428 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:10.707513 31485 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:10.708943 31428 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:10.714172 31549 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:10.715873 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 136920us
I20251025 20:04:10.671005 31793 consensus_queue.cc:1048] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20251025 20:04:10.719568 31780 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.776312 31780 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.690294 31645 consensus_queue.cc:1048] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251025 20:04:10.602489 31743 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.721879 31794 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.644577 30603 raft_consensus.cc:1275] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:04:10.786116 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:10.676082 30678 raft_consensus.cc:1275] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:04:10.735927 30719 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
W20251025 20:04:10.677083 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 176281us
I20251025 20:04:10.676165 30744 raft_consensus.cc:1240] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: []
W20251025 20:04:10.675470 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 177032us
I20251025 20:04:10.750082 30598 raft_consensus.cc:1275] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:10.743330 31801 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.751400 31614 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:10.805526 31794 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.806509 31801 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.807653 31290 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:10.808120 31794 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:10.774056 31732 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.751819 30750 raft_consensus.cc:1275] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 20:04:10.765502 30751 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:10.773347 30601 raft_consensus.cc:1275] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251025 20:04:10.770218 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.790616 31751 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.765645 31744 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:10.751632 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:10.810107 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 74273.7us
I20251025 20:04:10.840442 30719 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:10.810691 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:10.848292 30602 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
W20251025 20:04:10.778970 30645 connection.cc:232] RPC call timeout handler was delayed by -1.5446906429999672s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1705s for a response
W20251025 20:04:10.849390 30645 connection.cc:232] RPC call timeout handler was delayed by -1.528490642999941s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
W20251025 20:04:10.849730 30645 connection.cc:232] RPC call timeout handler was delayed by -1.5194906429999264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1733s for a response
I20251025 20:04:10.822860 31743 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:10.822647 31428 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:10.825052 31751 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.826062 31801 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:10.826411 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.884935 30598 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
W20251025 20:04:10.828176 31799 log.cc:927] Time spent T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.139s user 0.000s sys 0.003s
W20251025 20:04:10.891651 30645 connection.cc:232] RPC call timeout handler was delayed by -1.1830427209999925s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3018s for a response
I20251025 20:04:10.898105 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:10.898855 30601 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:10.904299 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 63971.8us
I20251025 20:04:10.905529 30719 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:10.907466 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:10.908456 30675 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:04:10.897653 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:10.894456 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:10.921017 30600 raft_consensus.cc:1275] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:10.922791 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:10.916229 31751 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:10.891785 31754 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:10.932183 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 133085us
I20251025 20:04:10.932368 30645 leader_election.cc:304] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:10.817281 31732 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.953018 30570 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:10.955879 30570 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:10.911504 31485 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.961685 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 627) took 1006 ms. Trace:
I20251025 20:04:10.965530 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:10.966595 30675 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:10.922874 31744 raft_consensus.cc:2804] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251025 20:04:10.952616 31721 consensus_meta.cc:287] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.726s user 0.015s sys 0.000s
I20251025 20:04:10.962889 30677 rpcz_store.cc:276] 1025 20:04:09.955130 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:10.009701 (+ 54571us) service_pool.cc:225] Handling call
1025 20:04:10.010716 (+ 1015us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:10.016427 (+ 5711us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:10.016488 (+ 61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:10.023472 (+ 6984us) log.cc:844] Serialized 26 byte log entry
1025 20:04:10.496630 (+473158us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:10.496673 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:10.496769 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:10.827569 (+330800us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:10.953033 (+125464us) raft_consensus.cc:1685] finished
1025 20:04:10.953067 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:10.961602 (+ 8535us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5042,"raft.queue_time_us":5239,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":7087,"spinlock_wait_cycles":4318464,"thread_start_us":389127,"threads_started":1,"wal-append.queue_time_us":513503}
W20251025 20:04:10.976301 31779 log.cc:927] Time spent T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20251025 20:04:10.929443 31679 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:10.928309 31628 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:10.951195 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:10.984257 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:10.986876 30675 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:10.966084 31624 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 7 FOLLOWER]: Leader election won for term 7
I20251025 20:04:10.967712 31696 raft_consensus.cc:2749] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:04:10.977787 31742 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.028s sys 0.041s
I20251025 20:04:10.970278 31744 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
I20251025 20:04:10.978507 31751 raft_consensus.cc:2749] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:11.000705 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.001678 30677 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:10.917826 31810 consensus_queue.cc:1048] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20251025 20:04:10.983235 31772 raft_consensus.cc:2749] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251025 20:04:11.008224 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:11.014380 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.015748 30677 raft_consensus.cc:2468] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:11.017197 31731 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:10.919065 31809 consensus_queue.cc:1048] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.162s
I20251025 20:04:11.006635 31744 raft_consensus.cc:3060] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:11.018549 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 86298.6us
W20251025 20:04:10.998430 31742 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s user 0.028s sys 0.042s
I20251025 20:04:10.970359 31485 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.027187 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.028414 30677 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:10.982560 31721 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:10.758235 30671 rpcz_store.cc:276] 1025 20:04:09.121283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:09.195177 (+ 73894us) service_pool.cc:225] Handling call
1025 20:04:09.195634 (+ 457us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:09.212292 (+ 16658us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:09.212331 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:09.212896 (+ 565us) log.cc:844] Serialized 26 byte log entry
1025 20:04:09.397173 (+184277us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:09.404496 (+ 7323us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:09.501825 (+ 97329us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:10.308613 (+806788us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:10.593251 (+284638us) raft_consensus.cc:1685] finished
1025 20:04:10.593290 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:10.674028 (+ 80738us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2179,"raft.queue_time_us":33271,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":41642,"spinlock_wait_cycles":9646336,"thread_start_us":157140,"threads_started":1,"wal-append.queue_time_us":340978}
I20251025 20:04:11.037788 30602 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:11.045557 30719 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:11.047256 30719 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:10.970054 31754 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.019397 31727 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:10.993059 31628 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.060715 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.028481 31499 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
@ 0x7fd81e116da4 google::LogMessage::SendToLog() at ??:0
I20251025 20:04:11.114192 31499 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.115669 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.116396 31499 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:11.042676 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.022423 31718 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.013077 30670 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
W20251025 20:04:11.117254 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.119478 31796 raft_consensus.cc:2804] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:11.119930 31796 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:11.019397 31751 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:11.120210 30675 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:11.094614 31628 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.094091 31808 consensus_queue.cc:1048] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
W20251025 20:04:11.096541 31814 log.cc:927] Time spent T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
W20251025 20:04:11.097323 31804 log.cc:927] Time spent T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.187s user 0.000s sys 0.000s
W20251025 20:04:11.099062 31815 log.cc:927] Time spent T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20251025 20:04:11.100944 31721 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:11.101315 31732 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.103490 31485 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:11.104027 30645 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:11.104561 31780 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:11.034329 31624 raft_consensus.cc:697] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 7 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:11.086051 31809 consensus_queue.cc:1048] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20251025 20:04:11.130883 31624 consensus_queue.cc:237] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.132253 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.030280 31604 consensus_queue.cc:1048] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.075s
I20251025 20:04:11.085041 31812 consensus_queue.cc:1048] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.250s
I20251025 20:04:11.093514 31752 consensus_queue.cc:1048] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.107s
I20251025 20:04:11.136353 30674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 628) took 1167 ms. Trace:
I20251025 20:04:11.105739 31641 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:11.107060 31813 log.cc:927] Time spent T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20251025 20:04:11.136729 30674 rpcz_store.cc:276] 1025 20:04:09.968777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:10.028595 (+ 59818us) service_pool.cc:225] Handling call
1025 20:04:10.028912 (+ 317us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:10.495038 (+466126us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:11.136095 (+641057us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b540017a7d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:11.136288 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":997200256}
I20251025 20:04:11.107255 31727 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.087621 31742 raft_consensus.cc:2804] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:11.110359 31754 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.110925 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.116545 30603 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:11.120307 31796 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:11.169688 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.170828 30603 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:11.172119 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.172832 30603 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:11.089211 30601 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:11.087679 30677 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:11.120541 31751 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.122797 31491 consensus_queue.cc:1048] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:11.124001 31686 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.113564 31428 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:04:11.142438 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 186661us
I20251025 20:04:11.149122 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.168951 30600 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:11.179914 30719 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:11.181036 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:11.181878 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.182854 30570 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:11.168306 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.184092 30601 raft_consensus.cc:3060] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:11.183813 30719 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:11.184381 30603 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:11.188120 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:11.193331 30719 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:11.194049 30600 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:11.195407 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.196442 30602 raft_consensus.cc:2468] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:11.198446 30719 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:11.199738 30719 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:11.172356 31731 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:11.203006 30570 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:11.205325 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.206280 30597 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
W20251025 20:04:11.209167 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 105238us
I20251025 20:04:11.176277 31742 raft_consensus.cc:697] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:11.185755 31428 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:11.140347 31641 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.195968 31686 raft_consensus.cc:2749] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:11.214160 30719 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:11.168813 31744 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.224136 31751 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.206367 31485 raft_consensus.cc:2804] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:11.224838 31485 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:11.225093 31744 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:11.225261 31485 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:11.183262 31721 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:11.219426 30570 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:11.226940 31721 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.212931 31812 consensus_queue.cc:1048] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20251025 20:04:11.229072 31721 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:11.223114 31499 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:11.230113 31499 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:11.230151 31754 raft_consensus.cc:2749] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:11.230607 31499 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:11.231215 31727 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.227306 31641 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:11.207523 31752 raft_consensus.cc:2804] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:11.201804 31797 raft_consensus.cc:2804] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:11.242636 31752 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:11.243139 31752 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:11.206426 31604 raft_consensus.cc:2804] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:11.206354 31809 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:11.243074 31797 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:11.221880 30719 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:11.274642 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 52877.6us
I20251025 20:04:11.244084 30602 raft_consensus.cc:1240] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:11.246663 30600 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:11.206293 31689 consensus_queue.cc:1048] T 53eca06f855d4e699f10ab84a2ba0981 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20251025 20:04:11.333199 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.386459 30601 raft_consensus.cc:2468] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:04:11.396265 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 175271us
W20251025 20:04:11.332531 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 113204us
I20251025 20:04:11.405761 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:11.273553 31809 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:04:11.408725 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 57456.7us
I20251025 20:04:11.406611 30678 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:11.415323 31800 consensus_queue.cc:1048] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:11.416149 31679 consensus_queue.cc:1048] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 20:04:11.370487 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.404871 30570 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:11.390781 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.383692 30719 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:11.402938 30596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 496) took 1296 ms. Trace:
I20251025 20:04:11.285552 31797 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:11.338119 31686 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:11.421510 30596 rpcz_store.cc:276] 1025 20:04:10.106466 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:10.106587 (+ 121us) service_pool.cc:225] Handling call
1025 20:04:11.332951 (+1226364us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:11.402828 (+ 69877us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3654400}
I20251025 20:04:11.423695 30719 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:11.405719 31721 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:11.423704 30678 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:11.333382 31754 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:11.298601 31793 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:11.288555 31780 raft_consensus.cc:2804] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:11.363338 31832 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:11.287212 31604 raft_consensus.cc:697] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:11.359241 31812 raft_consensus.cc:493] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:11.449513 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:11.454347 30719 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:11.463450 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.419560 31809 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:11.427857 31744 raft_consensus.cc:2804] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:11.463488 31754 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.477387 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:11.479591 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:11.466051 30750 raft_consensus.cc:1275] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:11.467944 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:11.468678 31823 log.cc:927] Time spent T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.451s user 0.002s sys 0.001s
I20251025 20:04:11.470204 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:11.470206 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:11.434386 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.506340 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.428562 31800 consensus_queue.cc:1048] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20251025 20:04:11.474254 30753 raft_consensus.cc:1275] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:11.430255 31686 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:11.504161 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.438437 31832 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.403810 31742 consensus_queue.cc:237] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.509840 30716 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
W20251025 20:04:11.472616 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 51718.3us
I20251025 20:04:11.439306 31780 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:04:11.519687 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.436540 31641 raft_consensus.cc:2804] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:11.462615 31628 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:11.482543 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:11.530396 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:11.483306 31478 raft_consensus.cc:2804] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:11.531409 30645 leader_election.cc:304] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:11.535027 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:11.540815 30745 raft_consensus.cc:1275] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:11.452287 31812 raft_consensus.cc:515] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.475981 31744 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:04:11.522341 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.542207 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.552038 30596 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:04:11.522020 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.519392 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:11.447978 31793 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.452999 31721 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.526443 31686 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:11.532232 31780 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:11.561920 31478 raft_consensus.cc:697] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:11.562937 31812 leader_election.cc:290] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.564028 31690 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:11.564180 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:11.551508 31641 raft_consensus.cc:697] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:11.565234 30645 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:11.545526 31811 log.cc:927] Time spent T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20251025 20:04:11.502419 31485 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.600628 31485 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:11.602116 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.571977 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:11.602907 30596 raft_consensus.cc:3060] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:11.602747 31641 consensus_queue.cc:237] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.545965 31794 raft_consensus.cc:2804] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251025 20:04:11.568065 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.555477 31744 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:11.616132 30751 raft_consensus.cc:2393] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:11.594980 31478 consensus_queue.cc:237] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.597859 30645 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:11.598594 31696 raft_consensus.cc:2804] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:11.608009 30744 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:11.609557 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.482784 31604 consensus_queue.cc:237] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.567847 31290 raft_consensus.cc:2749] T 775d2ddde3b44056818240b2a8402e9c P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:11.637363 30600 raft_consensus.cc:1275] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251025 20:04:11.638571 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.605775 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.612751 31794 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:11.694001 31626 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:11.683895 30596 raft_consensus.cc:2468] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:11.703306 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.704279 30678 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:11.709143 31290 raft_consensus.cc:2749] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:11.648761 31696 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:11.582316 31690 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.696851 31794 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:11.726555 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:11.715258 31823 log.cc:927] Time spent T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
W20251025 20:04:11.718252 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.720448 31696 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:11.712858 30671 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:11.678150 31428 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.772111 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.772991 30678 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:11.710141 31626 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.797783 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:11.806140 30678 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:11.812598 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:11.770915 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:11.823364 31752 consensus_meta.cc:287] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.580s user 0.006s sys 0.006s
I20251025 20:04:11.824386 31752 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.826372 30748 raft_consensus.cc:2393] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 5: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:11.827545 31752 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:11.835089 31832 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.836452 31793 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:11.838953 31841 log.cc:927] Time spent T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.137s user 0.000s sys 0.001s
I20251025 20:04:11.847653 30672 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
@ 0x7fd81e117950 google::LogMessage::Flush() at ??:0
I20251025 20:04:11.887615 31690 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.890048 31721 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:11.890797 31731 consensus_meta.cc:287] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.681s user 0.006s sys 0.007s
I20251025 20:04:11.891255 31731 raft_consensus.cc:2749] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:11.891763 31428 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:11.892050 30674 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:11.893397 31626 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:11.849375 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:11.900475 31499 consensus_meta.cc:287] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.669s user 0.015s sys 0.000s
I20251025 20:04:11.901078 31499 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.869517 30671 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:11.931165 30599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 649) took 1471 ms. Trace:
I20251025 20:04:11.935266 30599 rpcz_store.cc:276] 1025 20:04:10.459970 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:10.734864 (+274894us) service_pool.cc:225] Handling call
1025 20:04:10.737611 (+ 2747us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:11.574414 (+836803us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:11.574464 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:11.574528 (+ 64us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:11.574580 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:11.930611 (+356031us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b54001f8454. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:11.930699 (+ 88us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:11.931017 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1906622336}
I20251025 20:04:11.942625 31614 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:11.943424 31614 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:11.946611 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:11.902789 31626 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:11.959709 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:11.960326 31842 log.cc:927] Time spent T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.457s user 0.000s sys 0.003s
I20251025 20:04:11.918102 31780 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:11.983244 31754 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:11.991952 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 655) took 1530 ms. Trace:
W20251025 20:04:11.997926 31839 log.cc:927] Time spent T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.526s user 0.001s sys 0.004s
I20251025 20:04:12.002748 30677 raft_consensus.cc:1275] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:12.016180 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:12.022826 30673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 642) took 1168 ms. Trace:
I20251025 20:04:11.992564 30598 rpcz_store.cc:276] 1025 20:04:10.461869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:10.952160 (+490291us) service_pool.cc:225] Handling call
1025 20:04:10.981844 (+ 29684us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:11.111143 (+129299us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:11.111501 (+ 358us) log.cc:844] Serialized 14 byte log entry
1025 20:04:11.220638 (+109137us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:11.220717 (+ 79us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:11.220782 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:11.991563 (+770781us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b5400051ed4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:11.991640 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:11.991862 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5262,"prepare.queue_time_us":1708,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":16489,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"spinlock_wait_cycles":296760576,"thread_start_us":103365,"threads_started":1,"wal-append.queue_time_us":363710}
I20251025 20:04:12.028824 30679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41088 (request call id 643) took 1162 ms. Trace:
I20251025 20:04:12.023494 30673 rpcz_store.cc:276] 1025 20:04:10.854647 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:10.913585 (+ 58938us) service_pool.cc:225] Handling call
1025 20:04:12.021730 (+1108145us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:12.022072 (+ 342us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:12.022345 (+ 273us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:12.022384 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:12.022446 (+ 62us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:12.022494 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:12.022573 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:12.022739 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55753984}
I20251025 20:04:12.029244 30679 rpcz_store.cc:276] 1025 20:04:10.854971 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:10.911692 (+ 56721us) service_pool.cc:225] Handling call
1025 20:04:12.015899 (+1104207us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b5800213998. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:12.017325 (+ 1426us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3257728}
I20251025 20:04:11.988659 31626 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:12.054165 30675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 657) took 1054 ms. Trace:
I20251025 20:04:12.038980 30753 raft_consensus.cc:1275] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251025 20:04:12.200595 30570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.072s sys 0.089s
W20251025 20:04:12.335762 30570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.077s sys 0.090s
I20251025 20:04:12.198382 30604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 684) took 1062 ms. Trace:
I20251025 20:04:12.336642 30604 rpcz_store.cc:276] 1025 20:04:11.135769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.403769 (+268000us) service_pool.cc:225] Handling call
1025 20:04:11.404117 (+ 348us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:11.404471 (+ 354us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:11.427774 (+ 23303us) log.cc:844] Serialized 14 byte log entry
1025 20:04:11.696317 (+268543us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:11.713476 (+ 17159us) log.cc:844] Serialized 26 byte log entry
1025 20:04:11.713645 (+ 169us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:11.713688 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:11.713764 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:11.974421 (+260657us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b48000027d8. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd826a47971 00007fd826a3ea37 00007fd826a3e8ca 00007fd826b35a0f 00007fd826b356d7 00007fd826b49cfc 00007fd826b49ca8 00007fd826b49c28 00007fd826b49be3 00007fd826b48e8c 00007fd825a86326
1025 20:04:12.179756 (+205335us) raft_consensus.cc:1685] finished
1025 20:04:12.198024 (+ 18268us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:12.198284 (+ 260us) inbound_call.cc:173] Queueing success response
1025 20:04:12.312217 (+113933us) spinlock_profiling.cc:243] Waited 9.88 ms on lock 0x7b48000027d8. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd826a47971 00007fd826a3ea37 00007fd826a3e8ca 00007fd826b35a0f 00007fd826b356d7 00007fd826b49cfc 00007fd826b49ca8 00007fd826b49c28 00007fd826b49be3 00007fd826b48e8c 00007fd825a86326
Metrics: {"mutex_wait_us":3798,"prepare.queue_time_us":19009,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":4950,"raft.run_cpu_time_us":7782,"raft.run_wall_time_us":1176128,"spinlock_wait_cycles":45751424,"thread_start_us":218085,"threads_started":1,"wal-append.queue_time_us":695162}
W20251025 20:04:12.159564 30716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.013s sys 0.029s
W20251025 20:04:12.156350 30567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.011s sys 0.011s
W20251025 20:04:12.339478 30716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.806s user 0.016s sys 0.032s
W20251025 20:04:12.339588 30567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s user 0.018s sys 0.014s
W20251025 20:04:12.339903 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 806189us
W20251025 20:04:12.340071 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 834627us
I20251025 20:04:12.340586 30716 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
W20251025 20:04:12.200503 30745 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.012s sys 0.008s
W20251025 20:04:12.343231 30745 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.765s user 0.014s sys 0.015s
I20251025 20:04:12.331233 30602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 681) took 1207 ms. Trace:
I20251025 20:04:12.345485 30602 rpcz_store.cc:276] 1025 20:04:11.123696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.369988 (+246292us) service_pool.cc:225] Handling call
1025 20:04:11.370398 (+ 410us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:11.391961 (+ 21563us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:11.392009 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:11.408516 (+ 16507us) log.cc:844] Serialized 26 byte log entry
1025 20:04:11.446308 (+ 37792us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:04:11.446360 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:11.446469 (+ 109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:11.959278 (+512809us) spinlock_profiling.cc:243] Waited 6.79 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:12.039785 (+ 80507us) raft_consensus.cc:1685] finished
1025 20:04:12.039810 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:12.331125 (+291315us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5954,"raft.queue_time_us":21231,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":9280,"spinlock_wait_cycles":15609728,"thread_start_us":23690,"threads_started":1,"wal-append.queue_time_us":80964}
W20251025 20:04:12.119473 31809 consensus_meta.cc:287] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.612s user 0.012s sys 0.001s
I20251025 20:04:12.332090 30675 rpcz_store.cc:276] 1025 20:04:10.999327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.129009 (+129682us) service_pool.cc:225] Handling call
1025 20:04:11.138787 (+ 9778us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:11.221551 (+ 82764us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:11.221602 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:11.395919 (+174317us) log.cc:844] Serialized 26 byte log entry
1025 20:04:11.547585 (+151666us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:11.547629 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:11.547705 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:12.053833 (+506128us) raft_consensus.cc:1685] finished
1025 20:04:12.053864 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:12.054080 (+ 216us) inbound_call.cc:173] Queueing success response
1025 20:04:12.176385 (+122305us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b48000d1958. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd826a47971 00007fd826a3ea37 00007fd826a3e8ca 00007fd826b35a0f 00007fd826b356d7 00007fd826b49cfc 00007fd826b49ca8 00007fd826b49c28 00007fd826b49be3 00007fd826b48e8c 00007fd825a86326
Metrics: {"mutex_wait_us":35165,"raft.queue_time_us":52930,"raft.run_cpu_time_us":14260,"raft.run_wall_time_us":983188,"spinlock_wait_cycles":7580032,"thread_start_us":162411,"threads_started":2,"wal-append.queue_time_us":625468}
W20251025 20:04:12.318168 31848 log.cc:927] Time spent T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.130s user 0.002s sys 0.000s
W20251025 20:04:12.085322 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:12.085002 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:12.317824 30672 raft_consensus.cc:1275] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251025 20:04:12.196972 31796 consensus_meta.cc:287] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 1.011s user 0.009s sys 0.008s
W20251025 20:04:12.035521 30747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.028s sys 0.078s
W20251025 20:04:12.121536 31797 consensus_meta.cc:287] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.688s user 0.016s sys 0.001s
I20251025 20:04:12.373502 30673 raft_consensus.cc:1275] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:12.373931 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:12.373566 30750 raft_consensus.cc:1275] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:12.377857 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:12.019977 31614 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:12.213950 31771 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.003s sys 0.014s
I20251025 20:04:12.036269 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:12.380752 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:12.396726 30603 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.886s user 0.021s sys 0.039s
W20251025 20:04:12.400125 30603 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.889s user 0.021s sys 0.039s
I20251025 20:04:12.397410 30751 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:12.146561 31780 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:12.333125 31499 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:12.437165 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:12.437609 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:12.439241 30567 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:12.440658 30567 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:12.440764 30570 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:12.441673 30567 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:12.443493 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:12.396196 30747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.894s user 0.029s sys 0.094s
I20251025 20:04:12.368193 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:12.037531 31794 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:12.408290 31771 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.818s user 0.008s sys 0.014s
I20251025 20:04:12.457201 30746 raft_consensus.cc:1275] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:04:12.447196 30570 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:12.449450 30673 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:12.036674 31686 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:12.450927 30716 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:04:12.397164 30749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.012s sys 0.012s
W20251025 20:04:12.436627 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:12.463624 30751 raft_consensus.cc:1275] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:12.371886 31809 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:12.435556 31812 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:12.435231 31769 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:12.375056 30556 catalog_manager.cc:6112] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Tablet e8c971a6d4854180b4d243fa79fbbc11 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) was not created within the allowed timeout. Replacing with a new tablet f69f57fe417f492d9b82ed651b9d70c1
W20251025 20:04:12.466217 30749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s user 0.013s sys 0.013s
I20251025 20:04:12.465725 30675 raft_consensus.cc:1275] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251025 20:04:12.470991 30600 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.030s sys 0.033s
W20251025 20:04:12.475865 30600 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s user 0.030s sys 0.033s
I20251025 20:04:12.451785 31679 raft_consensus.cc:2804] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:12.447338 31801 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:12.447036 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:12.451467 31690 raft_consensus.cc:2749] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:12.408686 31796 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:12.473927 30719 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.018s user 0.016s sys 0.052s
W20251025 20:04:12.479616 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:12.454614 31772 raft_consensus.cc:2804] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:12.408541 31797 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:12.489248 30719 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.033s user 0.016s sys 0.052s
W20251025 20:04:12.474460 31744 consensus_meta.cc:287] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.804s user 0.015s sys 0.000s
W20251025 20:04:12.495220 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 1.03955e+06us
I20251025 20:04:12.454797 31732 consensus_queue.cc:1048] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20251025 20:04:12.497996 31610 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:12.455150 31754 consensus_queue.cc:1048] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
W20251025 20:04:12.505605 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:12.482992 31679 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:12.504190 30500 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39160 (request call id 94) took 4731 ms. Trace:
I20251025 20:04:12.485374 31751 raft_consensus.cc:2749] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:12.481979 31567 raft_consensus.cc:2804] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:12.516983 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:12.502192 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:12.524617 30556 catalog_manager.cc:6112] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Tablet fea0c74c9c0c4a69afe072c45c4cbe99 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) was not created within the allowed timeout. Replacing with a new tablet 7294f698d29e4c919745b460e4e25eeb
I20251025 20:04:12.491216 31772 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:12.535218 31772 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:12.509420 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:12.512610 31414 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.007s sys 0.035s
W20251025 20:04:12.538411 31414 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.576s user 0.007s sys 0.036s
I20251025 20:04:12.538630 31628 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:12.539230 31628 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:12.487638 31792 consensus_queue.cc:1048] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20251025 20:04:12.532783 30567 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:12.546862 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 46083.1us
I20251025 20:04:12.482733 31769 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:12.541642 31628 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:12.518615 31679 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:12.484308 31771 consensus_queue.cc:1048] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20251025 20:04:12.522015 31567 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 20:04:12.548631 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 64569.4us
I20251025 20:04:12.510694 30500 rpcz_store.cc:276] 1025 20:04:07.761984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:07.762182 (+ 198us) service_pool.cc:225] Handling call
1025 20:04:12.493401 (+4731219us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 20:04:10.023154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 20:04:10.124446 (+101292us) write_op.cc:432] Acquiring schema lock in shared mode
1025 20:04:10.134746 (+ 10300us) write_op.cc:435] Acquired schema lock
1025 20:04:10.137774 (+ 3028us) tablet.cc:660] Decoding operations
1025 20:04:10.287792 (+150018us) write_op.cc:620] Acquiring the partition lock for write op
1025 20:04:10.287913 (+ 121us) write_op.cc:641] Partition lock acquired for write op
1025 20:04:10.287947 (+ 34us) tablet.cc:683] Acquiring locks for 7 operations
1025 20:04:10.322214 (+ 34267us) tablet.cc:699] Row locks acquired
1025 20:04:10.322240 (+ 26us) write_op.cc:260] PREPARE: finished
1025 20:04:10.322436 (+ 196us) write_op.cc:270] Start()
1025 20:04:10.322614 (+ 178us) write_op.cc:276] Timestamp: P: 1761422650322368 usec, L: 0
1025 20:04:10.322639 (+ 25us) op_driver.cc:348] REPLICATION: starting
1025 20:04:10.442457 (+119818us) log.cc:844] Serialized 2890 byte log entry
1025 20:04:10.984202 (+541745us) op_driver.cc:464] REPLICATION: finished
1025 20:04:11.286683 (+302481us) write_op.cc:301] APPLY: starting
1025 20:04:11.289351 (+ 2668us) tablet.cc:1365] starting BulkCheckPresence
1025 20:04:11.403715 (+114364us) tablet.cc:1368] finished BulkCheckPresence
1025 20:04:11.431440 (+ 27725us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 20:04:11.449807 (+ 18367us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 20:04:11.450157 (+ 350us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
1025 20:04:11.450233 (+ 76us) write_op.cc:312] APPLY: finished
1025 20:04:11.450647 (+ 414us) log.cc:844] Serialized 58 byte log entry
1025 20:04:11.450996 (+ 349us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:04:11.451183 (+ 187us) write_op.cc:454] Released schema lock
1025 20:04:11.451263 (+ 80us) write_op.cc:341] FINISH: Updating metrics
1025 20:04:11.499506 (+ 48243us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:04:11.499546 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":665,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":283490,"apply.run_cpu_time_us":4601,"apply.run_wall_time_us":200423,"num_ops":7,"prepare.queue_time_us":146219,"prepare.run_cpu_time_us":33279,"prepare.run_wall_time_us":718178,"raft.queue_time_us":160490,"raft.run_cpu_time_us":68603,"raft.run_wall_time_us":282056,"replication_time_us":644268,"thread_start_us":637331,"threads_started":4,"wal-append.queue_time_us":307661}]]}
I20251025 20:04:12.486858 31801 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:12.487999 31812 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:12.531972 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:12.552099 30719 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:12.505125 31744 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:12.563836 30719 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:12.560719 30645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.925s user 0.013s sys 0.031s
W20251025 20:04:12.566155 30645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.931s user 0.013s sys 0.032s
W20251025 20:04:12.563185 31644 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.149s user 0.022s sys 0.040s
W20251025 20:04:12.569581 31644 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.156s user 0.023s sys 0.040s
I20251025 20:04:12.569672 31485 raft_consensus.cc:2804] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
W20251025 20:04:12.570853 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 973037us
I20251025 20:04:12.572650 30749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 419) took 1107 ms. Trace:
I20251025 20:04:12.573124 30749 rpcz_store.cc:276] 1025 20:04:11.473210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.506065 (+ 32855us) service_pool.cc:225] Handling call
1025 20:04:11.530572 (+ 24507us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:11.531076 (+ 504us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:11.531115 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:11.582614 (+ 51499us) log.cc:844] Serialized 26 byte log entry
1025 20:04:12.485761 (+903147us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:12.491804 (+ 6043us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:12.503374 (+ 11570us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:12.572356 (+ 68982us) raft_consensus.cc:1685] finished
1025 20:04:12.572386 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:12.572558 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10424,"raft.queue_time_us":3099,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":10639,"thread_start_us":764291,"threads_started":1,"wal-append.queue_time_us":924312}
W20251025 20:04:12.572762 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:04:12.572089 30645 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:12.577296 30645 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:12.578673 30556 catalog_manager.cc:6112] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Tablet da8a0eaa68044835a4cf94340ca25ac5 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) was not created within the allowed timeout. Replacing with a new tablet 01d56eb384124dfcb1bcd35d77e89f0e
I20251025 20:04:12.549526 31751 raft_consensus.cc:2749] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:12.562211 31567 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
W20251025 20:04:12.580039 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:12.556502 31604 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.708s user 0.004s sys 0.035s
I20251025 20:04:12.564723 31626 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:12.566584 31849 raft_consensus.cc:2804] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251025 20:04:12.627964 30678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.019s sys 0.020s
W20251025 20:04:12.636121 30678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s user 0.019s sys 0.020s
I20251025 20:04:12.637529 30673 raft_consensus.cc:1275] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:12.640477 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:12.627013 31718 raft_consensus.cc:2804] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:12.646672 31718 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:12.647085 31718 raft_consensus.cc:3060] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:04:12.621423 31478 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.818s user 0.017s sys 0.025s
I20251025 20:04:12.582659 31268 consensus_queue.cc:1048] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.055s
W20251025 20:04:12.681358 31478 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s user 0.017s sys 0.025s
W20251025 20:04:12.623545 31604 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s user 0.005s sys 0.035s
I20251025 20:04:12.652582 31849 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:12.699347 31849 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:12.633222 31754 consensus_queue.cc:1048] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20251025 20:04:12.702744 31610 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:12.655380 31732 consensus_queue.cc:1048] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20251025 20:04:12.703343 31610 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:12.704614 31732 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:12.705193 31732 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:12.707448 31732 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:12.611970 31847 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:12.707463 30556 catalog_manager.cc:6112] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Tablet 8e7a1b97c4aa4812a26b178e32476392 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) was not created within the allowed timeout. Replacing with a new tablet d8afaad4899545a9a4169026b63c0699
I20251025 20:04:12.655594 31628 consensus_queue.cc:1048] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20251025 20:04:12.626842 31850 consensus_queue.cc:1048] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
W20251025 20:04:12.633874 30595 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34886 (request call id 500) took 2461 ms (client timeout 2152 ms). Trace:
I20251025 20:04:12.710690 31801 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:12.628476 31742 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.009s sys 0.031s
W20251025 20:04:12.732231 30676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.892s user 0.022s sys 0.052s
W20251025 20:04:12.736874 30676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.896s user 0.023s sys 0.052s
W20251025 20:04:12.722790 30595 rpcz_store.cc:269] 1025 20:04:10.167280 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:10.171554 (+ 4274us) service_pool.cc:225] Handling call
1025 20:04:12.628237 (+2456683us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b5400307fd4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:12.628425 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5060614016}
I20251025 20:04:12.714651 31794 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:12.702752 31754 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:12.739952 31742 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s user 0.009s sys 0.032s
I20251025 20:04:12.734278 31809 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:12.760809 31696 consensus_meta.cc:287] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.948s user 0.014s sys 0.002s
I20251025 20:04:12.748948 31686 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:12.752187 31796 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:12.810323 30752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.802s user 0.028s sys 0.031s
W20251025 20:04:12.810847 30752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.803s user 0.028s sys 0.031s
I20251025 20:04:12.724166 31847 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:12.812383 31641 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.825s user 0.012s sys 0.048s
I20251025 20:04:12.742334 31772 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:12.839942 30556 catalog_manager.cc:6112] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Tablet 3f1269e0f1364a69b9a36ce877b0da5e (table test-workload [id=0622ca5c803d479d9858172423e767c7]) was not created within the allowed timeout. Replacing with a new tablet 6164729145ad47d6bbc2e7b3c6b49a22
W20251025 20:04:12.841879 30556 catalog_manager.cc:6112] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Tablet 4614bdb7f750431fa840cabccf782b4b (table test-workload [id=0622ca5c803d479d9858172423e767c7]) was not created within the allowed timeout. Replacing with a new tablet 8360e93e9b6d471791c2bd59bf8292ab
W20251025 20:04:12.843914 30556 catalog_manager.cc:6112] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Tablet 0d9831caf5d44fc080590d41bd68a083 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) was not created within the allowed timeout. Replacing with a new tablet 5ee1ba6e04b74e1faf5c9819144cd740
W20251025 20:04:12.845407 30556 catalog_manager.cc:6112] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Tablet 580e75e8dcd949e39aff92ada50b40be (table test-workload [id=0622ca5c803d479d9858172423e767c7]) was not created within the allowed timeout. Replacing with a new tablet 2d6dd59f914c41758a18803991215caf
W20251025 20:04:12.855981 31641 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.868s user 0.012s sys 0.049s
I20251025 20:04:12.762486 31754 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:12.805760 31696 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
W20251025 20:04:12.862257 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:12.814962 31855 raft_consensus.cc:2749] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:04:12.882853 31851 log.cc:927] Time spent T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.140s user 0.000s sys 0.003s
I20251025 20:04:12.884860 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 685) took 1746 ms. Trace:
W20251025 20:04:12.885390 31854 log.cc:927] Time spent T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.262s user 0.000s sys 0.001s
I20251025 20:04:12.885414 30603 rpcz_store.cc:276] 1025 20:04:11.138950 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.407099 (+268149us) service_pool.cc:225] Handling call
1025 20:04:11.407469 (+ 370us) raft_consensus.cc:1471] Updating replica for 3 ops
1025 20:04:11.427942 (+ 20473us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:11.427979 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 3 ops
1025 20:04:11.428723 (+ 744us) log.cc:844] Serialized 78 byte log entry
1025 20:04:12.400541 (+971818us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:12.451639 (+ 51098us) log.cc:844] Serialized 14 byte log entry
1025 20:04:12.481690 (+ 30051us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:12.481795 (+ 105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:12.882235 (+400440us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:12.884376 (+ 2141us) raft_consensus.cc:1685] finished
1025 20:04:12.884538 (+ 162us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:12.884764 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52568,"prepare.queue_time_us":464266,"raft.queue_time_us":54622,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":1910,"spinlock_wait_cycles":40005504,"thread_start_us":889797,"threads_started":1,"wal-append.queue_time_us":1159407}
W20251025 20:04:12.889325 31857 log.cc:927] Time spent T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20251025 20:04:12.891269 30600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 690) took 1464 ms. Trace:
I20251025 20:04:12.866954 31744 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:12.891788 30600 rpcz_store.cc:276] 1025 20:04:11.426718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.668763 (+242045us) service_pool.cc:225] Handling call
1025 20:04:11.677032 (+ 8269us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:11.677494 (+ 462us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:11.677538 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:11.697831 (+ 20293us) log.cc:844] Serialized 26 byte log entry
1025 20:04:12.477634 (+779803us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:12.477698 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:12.477817 (+ 119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:12.890904 (+413087us) raft_consensus.cc:1685] finished
1025 20:04:12.890939 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:12.891174 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":673,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":10282,"thread_start_us":717540,"threads_started":1,"wal-append.queue_time_us":1020689}
I20251025 20:04:12.871793 31769 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:12.903789 31855 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:12.918167 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 442638us
W20251025 20:04:12.944274 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 481768us
I20251025 20:04:12.911963 31797 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:12.956476 30674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.026s sys 0.015s
W20251025 20:04:12.957118 30674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s user 0.027s sys 0.015s
I20251025 20:04:12.959319 30570 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:12.961776 30570 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:12.969117 31772 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:12.971025 31754 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:12.972255 30671 consensus_meta.cc:287] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 1.053s user 0.011s sys 0.002s
I20251025 20:04:12.973008 30671 raft_consensus.cc:1275] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:12.951661 31855 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
@ 0x7fd81e11ca8b google::LogMessageFatal::~LogMessageFatal() at ??:0
W20251025 20:04:12.973868 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:12.990623 31847 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:12.914330 31812 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:12.992379 31732 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:12.995586 31679 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:12.996259 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:13.001955 30745 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.030s sys 0.062s
W20251025 20:04:13.002486 30745 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.568s user 0.030s sys 0.062s
W20251025 20:04:13.003774 30677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.015s sys 0.027s
W20251025 20:04:13.004200 30677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.015s sys 0.027s
W20251025 20:04:13.008268 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 53837.3us
W20251025 20:04:13.013305 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 48826.5us
W20251025 20:04:12.987839 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:12.967470 31769 raft_consensus.cc:2749] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251025 20:04:13.006603 31858 log.cc:927] Time spent T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
I20251025 20:04:12.977474 31855 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.000785 31696 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Requested vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717)
W20251025 20:04:13.015208 31860 log.cc:927] Time spent T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20251025 20:04:12.997846 31679 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:13.014655 30747 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.016s sys 0.044s
I20251025 20:04:13.029106 30716 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:13.039847 30570 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
W20251025 20:04:13.023008 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:12.986974 31718 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:13.027475 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.044448 30745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 424) took 1503 ms. Trace:
I20251025 20:04:13.016037 30671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 403) took 1480 ms. Trace:
I20251025 20:04:13.052259 30570 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:13.044013 30747 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.553s user 0.016s sys 0.044s
I20251025 20:04:13.017146 31812 consensus_queue.cc:1048] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
W20251025 20:04:13.056176 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 486045us
I20251025 20:04:13.022243 31485 raft_consensus.cc:697] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:13.047505 30745 rpcz_store.cc:276] 1025 20:04:11.541258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.541723 (+ 465us) service_pool.cc:225] Handling call
1025 20:04:11.555278 (+ 13555us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:11.566053 (+ 10775us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:11.566088 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:11.577740 (+ 11652us) log.cc:844] Serialized 26 byte log entry
1025 20:04:12.343757 (+766017us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:13.003013 (+659256us) log.cc:844] Serialized 14 byte log entry
1025 20:04:13.044087 (+ 41074us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:13.044191 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:13.044239 (+ 48us) raft_consensus.cc:1685] finished
1025 20:04:13.044254 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:13.044392 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31580,"prepare.queue_time_us":410406,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":37252,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":9355,"thread_start_us":1333794,"threads_started":2,"wal-append.queue_time_us":1041733}
I20251025 20:04:13.046923 31780 raft_consensus.cc:2749] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:13.053565 31679 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:13.022033 31610 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:13.061617 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 484365us
I20251025 20:04:13.063123 30570 leader_election.cc:304] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.046303 31771 consensus_queue.cc:1048] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20251025 20:04:13.071621 30570 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.056013 31644 raft_consensus.cc:2749] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:04:13.062021 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.051012 30671 rpcz_store.cc:276] 1025 20:04:11.535317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.868876 (+333559us) service_pool.cc:225] Handling call
1025 20:04:11.869334 (+ 458us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:12.973387 (+1104053us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:13.013825 (+ 40438us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:13.083173 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:13.083927 31862 log.cc:927] Time spent T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20251025 20:04:13.084098 30671 raft_consensus.cc:2468] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
W20251025 20:04:13.088119 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.061007 31679 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:13.088687 31679 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:13.091674 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.085490 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:13.050788 31567 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.095202 30676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 392) took 1565 ms. Trace:
I20251025 20:04:13.069872 31808 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:13.107462 31808 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:13.108021 31808 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:13.108530 31644 consensus_queue.cc:1048] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:04:13.103106 30676 rpcz_store.cc:276] 1025 20:04:11.529814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.837814 (+308000us) service_pool.cc:225] Handling call
1025 20:04:11.838296 (+ 482us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:11.838603 (+ 307us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:11.840155 (+ 1552us) log.cc:844] Serialized 14 byte log entry
1025 20:04:13.094563 (+1254408us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:13.094644 (+ 81us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:13.094708 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:13.094878 (+ 170us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:13.095122 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":196,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":299983,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":5623,"thread_start_us":1253538,"threads_started":2,"wal-append.queue_time_us":1163801}
W20251025 20:04:13.110534 30679 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.014s sys 0.038s
W20251025 20:04:13.110921 30679 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s user 0.014s sys 0.038s
I20251025 20:04:13.111181 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:13.064514 31754 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:13.112408 30645 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.111814 30676 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:13.062201 31847 consensus_queue.cc:1048] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20251025 20:04:13.076290 31772 consensus_queue.cc:1048] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
W20251025 20:04:13.144343 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 52303.1us
I20251025 20:04:13.134835 30752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 421) took 1626 ms. Trace:
I20251025 20:04:13.158254 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:13.161062 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:13.114359 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:13.162873 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 405) took 1627 ms. Trace:
I20251025 20:04:13.214599 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:13.157351 30745 raft_consensus.cc:1240] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251025 20:04:13.217543 30676 raft_consensus.cc:2468] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:13.217182 30750 raft_consensus.cc:2393] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 5: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:13.215373 30677 rpcz_store.cc:276] 1025 20:04:11.535851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:12.003686 (+467835us) service_pool.cc:225] Handling call
1025 20:04:12.004129 (+ 443us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:12.008655 (+ 4526us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:12.008706 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:12.335238 (+326532us) log.cc:844] Serialized 26 byte log entry
1025 20:04:13.004651 (+669413us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:13.004727 (+ 76us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:13.004819 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:13.162625 (+157806us) raft_consensus.cc:1685] finished
1025 20:04:13.162669 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:13.162815 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13552,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":15375,"thread_start_us":569534,"threads_started":1,"wal-append.queue_time_us":772501}
I20251025 20:04:13.218546 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.219187 30677 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:13.146905 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 49615.3us
I20251025 20:04:13.214552 30752 rpcz_store.cc:276] 1025 20:04:11.508762 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.508937 (+ 175us) service_pool.cc:225] Handling call
1025 20:04:12.006345 (+497408us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:12.006625 (+ 280us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:12.007149 (+ 524us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:12.007192 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:12.007911 (+ 719us) log.cc:844] Serialized 26 byte log entry
1025 20:04:12.811398 (+803487us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:04:12.811462 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:12.811597 (+ 135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:13.110066 (+298469us) raft_consensus.cc:1685] finished
1025 20:04:13.110089 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:13.134744 (+ 24655us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11285,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":12739,"spinlock_wait_cycles":3574144,"thread_start_us":802945,"threads_started":1,"wal-append.queue_time_us":1033100}
I20251025 20:04:13.156565 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:13.149916 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:13.216804 30675 raft_consensus.cc:2393] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:13.221340 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.276014 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:13.106523 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:13.282371 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 188596us
I20251025 20:04:13.281670 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:13.141250 31846 log.cc:927] Time spent T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20251025 20:04:13.091284 31871 raft_consensus.cc:2749] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:13.283645 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.110854 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:13.286367 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:13.297933 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:13.095567 31788 raft_consensus.cc:493] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:13.220206 31610 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:13.157488 31865 log.cc:927] Time spent T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.175s user 0.002s sys 0.000s
I20251025 20:04:13.110648 31686 raft_consensus.cc:2749] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:13.113380 31690 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:13.162505 31867 log.cc:927] Time spent T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20251025 20:04:13.156225 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:13.140443 30747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 409) took 1707 ms. Trace:
I20251025 20:04:13.304955 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:13.317763 30750 raft_consensus.cc:2393] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
W20251025 20:04:13.324944 31870 log.cc:927] Time spent T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.238s user 0.001s sys 0.001s
I20251025 20:04:13.328328 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:13.272966 31812 consensus_queue.cc:1048] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20251025 20:04:13.332244 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:13.329679 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.307646 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:13.134408 31849 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.152969 30671 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:13.221307 31751 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:13.334082 30676 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:13.135076 31628 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:13.314244 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 100046us
I20251025 20:04:13.290381 31801 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:13.336859 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 225929us
I20251025 20:04:13.339025 30567 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:13.272418 31754 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.342043 30677 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:13.343197 30567 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:04:13.317776 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.345571 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:13.346323 30677 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:13.348079 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.348996 30675 raft_consensus.cc:2393] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:13.348440 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.313585 30672 raft_consensus.cc:2468] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:13.350159 30567 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:13.350664 30645 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.187556 31485 consensus_queue.cc:237] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.336120 31855 raft_consensus.cc:2749] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:13.342564 31628 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.338796 31718 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.356176 31567 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.345382 31769 raft_consensus.cc:2804] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:13.371521 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.373071 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:13.318001 31690 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.374398 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.376560 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.309036 31788 raft_consensus.cc:515] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.379902 30678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 671) took 1876 ms. Trace:
I20251025 20:04:13.353546 31772 raft_consensus.cc:2804] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:13.380548 30602 raft_consensus.cc:2393] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:13.381708 30595 raft_consensus.cc:2393] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 5: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:13.374352 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:13.385674 30678 rpcz_store.cc:276] 1025 20:04:11.503147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.819090 (+315943us) service_pool.cc:225] Handling call
1025 20:04:11.858564 (+ 39474us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:11.859112 (+ 548us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:11.859160 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:12.047100 (+187940us) log.cc:844] Serialized 26 byte log entry
1025 20:04:12.636595 (+589495us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:12.636661 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:12.636764 (+ 103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:13.156824 (+520060us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:13.364072 (+207248us) raft_consensus.cc:1685] finished
1025 20:04:13.364111 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:13.379817 (+ 15706us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":194,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":136,"spinlock_wait_cycles":38870912,"thread_start_us":516574,"threads_started":1,"wal-append.queue_time_us":882251}
I20251025 20:04:13.394546 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:13.395545 30599 raft_consensus.cc:2393] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:13.395737 30598 raft_consensus.cc:1275] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251025 20:04:13.396433 30672 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:13.397799 30672 raft_consensus.cc:3055] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 6 LEADER]: Stepping down as leader of term 6
I20251025 20:04:13.398175 31690 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.398191 30672 raft_consensus.cc:740] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:13.398852 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.399282 31628 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.350165 31801 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.399605 30672 consensus_queue.cc:260] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.399636 30671 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:13.401085 30679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 414) took 1050 ms. Trace:
I20251025 20:04:13.401510 30679 rpcz_store.cc:276] 1025 20:04:12.350239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:12.448342 (+ 98103us) service_pool.cc:225] Handling call
1025 20:04:12.464589 (+ 16247us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:12.471758 (+ 7169us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:12.471811 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:12.489854 (+ 18043us) log.cc:844] Serialized 26 byte log entry
1025 20:04:13.111288 (+621434us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:04:13.111332 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:13.111398 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:13.400876 (+289478us) raft_consensus.cc:1685] finished
1025 20:04:13.400902 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:13.401024 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12308,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":6098,"thread_start_us":545662,"threads_started":1,"wal-append.queue_time_us":796922}
I20251025 20:04:13.405412 30679 raft_consensus.cc:3055] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:13.406126 30672 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Advancing to term 7
W20251025 20:04:13.407027 31874 log.cc:927] Time spent T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20251025 20:04:13.409210 31754 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.410372 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.411011 30604 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:13.413340 31732 raft_consensus.cc:2749] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:13.339031 31751 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.362005 31850 raft_consensus.cc:2804] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:13.416924 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:13.405732 30679 raft_consensus.cc:740] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:13.427896 30645 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.435016 30679 consensus_queue.cc:260] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.425812 30604 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:13.437207 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.438062 30604 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:13.446130 30601 raft_consensus.cc:2468] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:13.365820 31769 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:13.455680 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.456682 30602 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:13.451821 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.452446 30679 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
W20251025 20:04:13.415278 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.414309 31679 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.461876 31679 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.466270 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:13.467715 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.468817 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.401510 31801 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.384423 31847 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:13.470347 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 78417.6us
I20251025 20:04:13.481284 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:13.481588 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.481964 30595 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:13.471619 31751 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.474109 31696 raft_consensus.cc:2749] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:13.449489 31850 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:13.483378 31850 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:13.474472 31847 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:13.485479 31847 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:13.416694 31772 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:13.486295 31772 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:13.317049 30747 rpcz_store.cc:276] 1025 20:04:11.433473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:11.462343 (+ 28870us) service_pool.cc:225] Handling call
1025 20:04:11.500343 (+ 38000us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:11.500874 (+ 531us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:11.500910 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:11.501502 (+ 592us) log.cc:844] Serialized 26 byte log entry
1025 20:04:12.457796 (+956294us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:13.059900 (+602104us) log.cc:844] Serialized 14 byte log entry
1025 20:04:13.080694 (+ 20794us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:13.095083 (+ 14389us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:13.095772 (+ 689us) raft_consensus.cc:1685] finished
1025 20:04:13.104711 (+ 8939us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:13.135773 (+ 31062us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27895,"prepare.queue_time_us":287281,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":27183,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":19421,"thread_start_us":1512309,"threads_started":2,"wal-append.queue_time_us":1124971}
I20251025 20:04:13.479108 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.488138 30670 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 5
I20251025 20:04:13.489987 31769 raft_consensus.cc:3060] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:13.449048 31882 raft_consensus.cc:2749] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:13.481828 30645 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.495841 31696 raft_consensus.cc:2749] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:13.469858 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.496738 30598 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:13.497153 31849 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:13.482642 30678 raft_consensus.cc:3055] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:13.503022 30678 raft_consensus.cc:740] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:13.504030 30678 consensus_queue.cc:260] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.505307 30678 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
W20251025 20:04:13.475940 31887 log.cc:927] Time spent T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20251025 20:04:13.480760 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.507097 30676 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 5
I20251025 20:04:13.510092 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:13.510474 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.511025 30600 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:13.510121 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.533747 30595 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:13.535923 30604 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
W20251025 20:04:13.536401 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 54671.8us
I20251025 20:04:13.413754 31788 leader_election.cc:290] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:13.546630 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:13.547320 30752 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:13.554795 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:13.555650 30595 raft_consensus.cc:1240] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251025 20:04:13.588086 30752 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
W20251025 20:04:13.588202 31809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.003s sys 0.015s
I20251025 20:04:13.537698 31847 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:13.589290 30570 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.556782 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:13.541850 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.601110 30600 raft_consensus.cc:2468] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
W20251025 20:04:13.548472 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.605304 30570 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.608448 30676 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:13.620036 30602 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:13.630412 30645 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.542250 31849 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:13.633965 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:13.597455 31809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s user 0.003s sys 0.016s
I20251025 20:04:13.543056 30671 raft_consensus.cc:2468] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:13.598759 31679 raft_consensus.cc:493] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:13.598349 31610 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:13.598448 31478 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:13.648223 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:13.651557 30678 raft_consensus.cc:1275] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20251025 20:04:13.655390 30678 pending_rounds.cc:85] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 1
I20251025 20:04:13.655747 30678 pending_rounds.cc:107] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251025 20:04:13.656054 30678 raft_consensus.cc:2889] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:04:13.546751 31808 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.661324 31808 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:13.662626 30748 raft_consensus.cc:1240] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251025 20:04:13.593623 31771 consensus_queue.cc:1059] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
W20251025 20:04:13.639030 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:13.639392 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.638618 31686 raft_consensus.cc:2749] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251025 20:04:13.652916 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 54496.3us
I20251025 20:04:13.627990 31880 raft_consensus.cc:2749] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251025 20:04:13.675366 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:13.676304 30752 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Advancing to term 6
W20251025 20:04:13.677626 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:04:13.678678 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:13.679682 30751 raft_consensus.cc:2393] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 5: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:13.682163 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:13.647575 31855 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:13.648836 31879 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:13.603128 31847 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.686372 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:13.689668 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.692363 30678 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 3
I20251025 20:04:13.650172 30670 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
W20251025 20:04:13.688254 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 61416.7us
I20251025 20:04:13.669267 31610 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.707772 31610 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:13.666364 31679 raft_consensus.cc:515] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.712198 31679 leader_election.cc:290] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:13.710124 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.647560 31849 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.717056 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:13.723332 30678 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:13.724776 30752 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 6.
I20251025 20:04:13.731015 30645 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.703576 31879 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.666754 31478 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.692796 31882 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:13.785396 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 88814.7us
I20251025 20:04:13.796716 30570 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:13.798247 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 108344us
W20251025 20:04:13.780572 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.770130 31718 raft_consensus.cc:2749] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:13.842746 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:13.843894 30599 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:13.847194 31847 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.780210 31882 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.863312 30598 raft_consensus.cc:2468] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:13.865072 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:13.867520 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 70865.1us
I20251025 20:04:13.868216 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:13.873589 30645 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:13.859503 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:13.870970 31908 log.cc:927] Time spent T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20251025 20:04:13.859840 30672 raft_consensus.cc:1275] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20251025 20:04:13.881546 30570 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:13.870774 31744 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:13.876233 31679 raft_consensus.cc:2804] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:13.884799 31849 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:13.903601 31679 raft_consensus.cc:697] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:13.884150 30672 pending_rounds.cc:85] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 3
I20251025 20:04:13.903610 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:13.881573 31696 raft_consensus.cc:2804] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:13.908510 30751 raft_consensus.cc:1240] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:13.908740 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:13.899487 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.910792 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:13.899705 31880 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election won for term 6
I20251025 20:04:13.907094 30672 pending_rounds.cc:107] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 6.4
W20251025 20:04:13.915493 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 61340.6us
I20251025 20:04:13.899871 31644 consensus_queue.cc:1059] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20251025 20:04:13.931342 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:13.910707 31696 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:13.927764 30746 raft_consensus.cc:1240] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251025 20:04:13.921144 30672 raft_consensus.cc:2889] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:04:13.907393 31744 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:13.946136 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:13.915994 31567 consensus_queue.cc:1059] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20251025 20:04:13.938052 31696 raft_consensus.cc:3060] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:13.963667 31679 consensus_queue.cc:237] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.967864 31772 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.976665 30603 raft_consensus.cc:1240] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:13.978809 31772 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:13.981405 31812 consensus_queue.cc:1059] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:13.963537 31879 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:13.986374 31902 log.cc:927] Time spent T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.204s user 0.001s sys 0.000s
W20251025 20:04:13.995168 31907 log.cc:927] Time spent T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20251025 20:04:13.995731 31899 log.cc:927] Time spent T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20251025 20:04:13.996860 31880 raft_consensus.cc:697] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 6 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:13.997956 31880 consensus_queue.cc:237] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:13.999680 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:14.000277 30601 raft_consensus.cc:1275] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:14.000329 30673 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:13.947391 31478 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:14.013128 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.015766 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:13.972465 31906 log.cc:927] Time spent T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
W20251025 20:04:14.008885 31850 consensus_meta.cc:287] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.525s user 0.014s sys 0.000s
I20251025 20:04:14.041304 30567 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.051604 30567 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:13.981283 31847 raft_consensus.cc:3055] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:14.043533 30716 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:14.054252 30716 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:14.044651 30601 raft_consensus.cc:1275] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:14.059471 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.003463 31644 raft_consensus.cc:3055] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:14.061010 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.066604 30676 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:14.063215 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.061774 30567 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.010700 31882 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:14.084357 30567 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.088663 30645 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.089565 30567 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.084537 30676 raft_consensus.cc:1275] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251025 20:04:14.065435 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.063658 31414 raft_consensus.cc:2764] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20251025 20:04:14.072263 31903 log.cc:927] Time spent T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.158s user 0.004s sys 0.001s
I20251025 20:04:14.063506 31732 raft_consensus.cc:2749] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:14.096012 30675 raft_consensus.cc:1240] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
W20251025 20:04:14.108705 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 76711.4us
I20251025 20:04:14.096957 30716 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:14.088135 31888 log.cc:927] Time spent T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20251025 20:04:14.087587 31904 log.cc:927] Time spent T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20251025 20:04:14.070497 31690 raft_consensus.cc:3060] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:14.114586 30716 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:14.110653 30675 raft_consensus.cc:1240] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:14.118711 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:14.086781 31885 raft_consensus.cc:2804] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:14.058909 31847 raft_consensus.cc:740] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:14.060693 31644 raft_consensus.cc:740] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:14.087973 31478 raft_consensus.cc:2804] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:14.093652 31686 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:14.086498 31751 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:14.094089 31879 raft_consensus.cc:2749] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:14.126449 31686 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:14.128368 31686 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
W20251025 20:04:14.129073 31911 log.cc:927] Time spent T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.208s user 0.001s sys 0.000s
I20251025 20:04:14.132210 31751 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
@ 0x7fd8280782ae kudu::TestWorkload::Setup() at ??:0
W20251025 20:04:14.135083 31910 log.cc:927] Time spent T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
W20251025 20:04:14.137051 31769 consensus_meta.cc:287] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.646s user 0.010s sys 0.000s
I20251025 20:04:14.138744 31751 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:14.140749 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:14.126102 31788 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:14.142221 31788 raft_consensus.cc:697] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:14.142576 31744 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:14.143344 31788 consensus_queue.cc:237] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:14.118606 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
@ 0x354cc8 kudu::master::LeaderRebalancerTest::CreateWorkloadTable() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_leader_rebalancer-test.cc:110
I20251025 20:04:14.094369 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
@ 0x350d2f kudu::master::LeaderRebalancerTest_AddTserver_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_leader_rebalancer-test.cc:415
I20251025 20:04:14.114042 31801 raft_consensus.cc:2804] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:14.147877 31801 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:14.148267 31801 raft_consensus.cc:3060] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:14.152122 31871 raft_consensus.cc:3055] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:14.152510 31871 raft_consensus.cc:740] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:14.062605 31850 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.167371 30673 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:14.122879 31491 raft_consensus.cc:2804] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:14.171978 31491 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:14.172276 30599 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:14.172425 31491 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:14.173568 30597 raft_consensus.cc:1240] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:14.173986 31850 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:14.175491 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 62853.4us
I20251025 20:04:14.175912 30719 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:04:14.082326 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.137457 31769 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.184863 30719 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:04:14.244606 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:14.185397 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 60028.6us
I20251025 20:04:14.153342 31871 consensus_queue.cc:260] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:14.247733 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 63025.7us
W20251025 20:04:14.251088 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 70298.2us
I20251025 20:04:14.253723 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.255656 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:14.242015 31910 log.cc:927] Time spent T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.106s user 0.000s sys 0.002s
I20251025 20:04:14.260771 31794 consensus_queue.cc:1048] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:04:14.262027 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:14.262526 30716 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:14.123732 31882 consensus_queue.cc:1048] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.123s
I20251025 20:04:14.256342 30673 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
W20251025 20:04:14.235502 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:14.237454 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.255671 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:14.169956 31885 raft_consensus.cc:697] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:14.299394 30596 raft_consensus.cc:1240] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:14.182755 31610 raft_consensus.cc:2749] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:14.307260 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.311285 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.312706 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.314790 30677 raft_consensus.cc:2393] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 4: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:14.245608 31754 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:14.254489 31626 raft_consensus.cc:2749] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:14.324398 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.328311 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:14.330346 30676 raft_consensus.cc:1275] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:14.309278 31871 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:14.324748 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.304828 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:14.399993 30645 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:14.411882 30678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.019s sys 0.024s
W20251025 20:04:14.412389 30678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s user 0.019s sys 0.024s
I20251025 20:04:14.313931 31718 consensus_queue.cc:1048] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
W20251025 20:04:14.411499 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.257711 31644 consensus_queue.cc:260] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.255007 31847 consensus_queue.cc:260] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:14.462762 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 200316us
I20251025 20:04:14.374634 31744 raft_consensus.cc:2804] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:14.323487 31754 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:14.474825 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 222880us
I20251025 20:04:14.469949 30716 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:14.476907 30716 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:14.455096 31855 consensus_meta.cc:287] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.761s user 0.017s sys 0.002s
I20251025 20:04:14.485165 30719 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:14.487758 30719 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
W20251025 20:04:14.472864 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:14.488809 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.473004 31912 raft_consensus.cc:2749] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:14.507143 30719 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:14.508358 30597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 701) took 1135 ms. Trace:
I20251025 20:04:14.508952 30596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 702) took 1135 ms. Trace:
I20251025 20:04:14.509385 30596 rpcz_store.cc:276] 1025 20:04:13.373078 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:13.383427 (+ 10349us) service_pool.cc:225] Handling call
1025 20:04:13.383877 (+ 450us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:14.305511 (+921634us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:14.508733 (+203222us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b5400041ad4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:14.508895 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2104692096}
I20251025 20:04:14.508756 30597 rpcz_store.cc:276] 1025 20:04:13.372810 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:13.372940 (+ 130us) service_pool.cc:225] Handling call
1025 20:04:13.373241 (+ 301us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:14.174003 (+800762us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:14.508059 (+334056us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b54000f0854. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:14.508284 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1840080128}
I20251025 20:04:14.511577 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.512234 30675 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:14.492393 31644 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:14.494248 31855 raft_consensus.cc:2749] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:14.494439 31876 raft_consensus.cc:2804] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:14.475080 31809 consensus_queue.cc:1048] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
W20251025 20:04:14.506299 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.544100 30719 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:14.545608 31849 raft_consensus.cc:2749] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:14.546378 31794 raft_consensus.cc:2804] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
W20251025 20:04:14.508553 31917 log.cc:927] Time spent T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20251025 20:04:14.503548 31610 raft_consensus.cc:2749] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251025 20:04:14.528728 31915 log.cc:927] Time spent T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20251025 20:04:14.491149 31744 raft_consensus.cc:697] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:14.563118 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 217554us
I20251025 20:04:14.528299 31892 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:14.530616 31769 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:14.577562 30674 raft_consensus.cc:3060] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
W20251025 20:04:14.583065 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 401171us
I20251025 20:04:14.584352 30604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 722) took 1179 ms. Trace:
I20251025 20:04:14.606667 30604 rpcz_store.cc:276] 1025 20:04:13.405326 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:13.600935 (+195609us) service_pool.cc:225] Handling call
1025 20:04:13.601271 (+ 336us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:13.601526 (+ 255us) raft_consensus.cc:1534] Early marking committed up to index 3
1025 20:04:13.602265 (+ 739us) log.cc:844] Serialized 14 byte log entry
1025 20:04:13.746724 (+144459us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b180031e1c0. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8278d09a1 00007fd8278dd371 00007fd81ea702f7 00007fd827849219 00007fd81ea8cc4b 00007fd81ea8aeef 00007fd81ea8d1b2 00007fd8216d42bb 00007fd8216d9dc3 00007fd8216da2da 00007fd821b445ba
1025 20:04:14.583819 (+837095us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:14.583893 (+ 74us) raft_consensus.cc:1645] Marking committed up to 3
1025 20:04:14.583958 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:14.584061 (+ 103us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:14.584259 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":337654,"prepare.queue_time_us":164,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":13866,"raft.queue_time_us":339850,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":2869,"spinlock_wait_cycles":41946880,"thread_start_us":499080,"threads_started":1,"wal-append.queue_time_us":748160}
I20251025 20:04:14.587896 30600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 723) took 1182 ms. Trace:
W20251025 20:04:14.594031 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.493269 31885 consensus_queue.cc:237] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:14.609187 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 65227.3us
I20251025 20:04:14.615039 30567 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.616662 30719 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:14.618311 30645 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.608649 30600 rpcz_store.cc:276] 1025 20:04:13.405599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:13.608253 (+202654us) service_pool.cc:225] Handling call
1025 20:04:13.608555 (+ 302us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:13.608750 (+ 195us) raft_consensus.cc:1534] Early marking committed up to index 3
1025 20:04:13.850384 (+241634us) log.cc:844] Serialized 14 byte log entry
1025 20:04:14.587529 (+737145us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:14.587582 (+ 53us) raft_consensus.cc:1645] Marking committed up to 3
1025 20:04:14.587626 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:14.587703 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:14.587825 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":275877,"prepare.queue_time_us":85705,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":2753,"raft.queue_time_us":245028,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":3613,"thread_start_us":708145,"threads_started":2,"wal-append.queue_time_us":646259}
I20251025 20:04:14.616354 31876 raft_consensus.cc:697] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:14.634560 31751 raft_consensus.cc:2749] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:14.643119 31876 consensus_queue.cc:237] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.645344 30745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 452) took 1110 ms. Trace:
I20251025 20:04:14.645850 30745 rpcz_store.cc:276] 1025 20:04:13.534573 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:13.534734 (+ 161us) service_pool.cc:225] Handling call
1025 20:04:13.989510 (+454776us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826af7580 00007fd826ae1ee9 00007fd826ae1992 00007fd82007bd34
1025 20:04:13.989754 (+ 244us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:13.990210 (+ 456us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:13.990247 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:13.990769 (+ 522us) log.cc:844] Serialized 26 byte log entry
1025 20:04:14.458392 (+467623us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:14.458456 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:14.458567 (+ 111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:14.644933 (+186366us) raft_consensus.cc:1685] finished
1025 20:04:14.644964 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:14.645257 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4830,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":123,"spinlock_wait_cycles":47916416,"thread_start_us":467141,"threads_started":1,"wal-append.queue_time_us":600903}
I20251025 20:04:14.650835 31478 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:14.651335 31478 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:14.638089 31855 consensus_queue.cc:1048] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
W20251025 20:04:14.652655 31686 consensus_meta.cc:287] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.524s user 0.009s sys 0.000s
I20251025 20:04:14.653167 31686 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:14.637959 31921 log.cc:927] Time spent T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20251025 20:04:14.638691 31628 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:14.636911 31922 log.cc:927] Time spent T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20251025 20:04:14.673262 30674 raft_consensus.cc:2468] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:14.624126 30567 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:14.675078 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 51257.1us
I20251025 20:04:14.676126 30567 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.677525 30567 leader_election.cc:304] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.681461 31812 raft_consensus.cc:2804] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:14.682093 31772 raft_consensus.cc:2749] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:14.682392 31939 raft_consensus.cc:2749] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:14.683708 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.684264 31936 consensus_queue.cc:1059] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:04:14.685339 31628 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.685963 31812 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:14.686403 31812 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:14.687604 31628 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:14.635742 31801 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:14.693591 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.693513 30567 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.695811 30567 leader_election.cc:304] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.702311 30567 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.657042 31892 raft_consensus.cc:697] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:14.666842 31809 raft_consensus.cc:2749] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251025 20:04:14.724334 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 106433us
I20251025 20:04:14.727269 30567 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.687628 31754 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:14.729220 30567 leader_election.cc:304] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.700901 31628 raft_consensus.cc:2749] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:14.703286 31788 raft_consensus.cc:3055] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:14.702663 31932 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:14.706434 31772 raft_consensus.cc:2749] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:14.725330 31941 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:14.711669 31942 raft_consensus.cc:2749] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:14.709198 31769 raft_consensus.cc:2749] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251025 20:04:14.739215 30645 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:14.751394 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 495308us
I20251025 20:04:14.738574 31751 raft_consensus.cc:2804] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251025 20:04:14.779137 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 45150.2us
I20251025 20:04:14.740857 31788 raft_consensus.cc:740] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:14.742471 31754 raft_consensus.cc:2749] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:04:14.785750 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 56649.2us
I20251025 20:04:14.745769 31940 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:14.753379 31923 log.cc:927] Time spent T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.201s user 0.000s sys 0.003s
W20251025 20:04:14.796320 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.798305 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.799824 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:14.801378 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 251369us
I20251025 20:04:14.803682 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.805074 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:14.750970 31794 raft_consensus.cc:697] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:14.780252 31855 raft_consensus.cc:2749] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:14.781409 31751 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:14.814592 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:14.815487 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:14.815086 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:14.831259 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 140042us
I20251025 20:04:14.745213 31744 consensus_queue.cc:237] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.817054 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:14.819476 31751 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:14.755895 31941 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:14.822852 31690 consensus_meta.cc:287] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.700s user 0.003s sys 0.008s
I20251025 20:04:14.841830 30716 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:14.847090 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.844908 30719 leader_election.cc:304] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:14.877375 30677 raft_consensus.cc:3055] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 5 LEADER]: Stepping down as leader of term 5
I20251025 20:04:14.877916 30677 raft_consensus.cc:740] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
W20251025 20:04:14.882380 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 84464.1us
W20251025 20:04:14.890763 31944 log.cc:927] Time spent T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20251025 20:04:14.842453 31788 consensus_queue.cc:260] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.892907 31788 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:14.893323 31847 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:14.894783 30645 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.899008 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:04:14.864403 31696 consensus_meta.cc:287] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.858s user 0.007s sys 0.007s
I20251025 20:04:14.862138 31690 raft_consensus.cc:2749] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:14.905655 31809 raft_consensus.cc:2804] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:14.906199 31809 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:14.906569 31809 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:14.822134 31892 consensus_queue.cc:237] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.807354 31940 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.911751 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:14.913712 30677 consensus_queue.cc:260] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.918846 30677 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:14.863124 31686 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:14.871929 30749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 448) took 1334 ms. Trace:
I20251025 20:04:14.944548 31732 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:14.945180 31732 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:14.947413 31769 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:14.948107 31769 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:14.915423 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:14.885934 31801 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:14.953850 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 70432.9us
I20251025 20:04:14.905119 31958 raft_consensus.cc:2749] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:14.882256 31797 consensus_queue.cc:1048] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.058s
W20251025 20:04:14.959379 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 84254.7us
W20251025 20:04:14.964340 31919 log.cc:927] Time spent T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.457s user 0.000s sys 0.005s
I20251025 20:04:14.934986 31855 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:14.957335 30567 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:14.952664 31732 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:14.971336 30719 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
W20251025 20:04:14.981598 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 151499us
I20251025 20:04:15.254077 31945 consensus_queue.cc:1048] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:04:15.254472 30716 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:14.984074 30719 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:15.254793 31788 consensus_queue.cc:1059] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:14.871495 31794 consensus_queue.cc:237] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.255757 31953 raft_consensus.cc:2804] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:14.989379 31941 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:14.991552 30751 raft_consensus.cc:1275] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251025 20:04:15.256307 31953 raft_consensus.cc:697] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:14.992148 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:15.257225 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:15.257683 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:15.258051 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:15.055292 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 96810.4us
I20251025 20:04:15.052323 31801 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:15.258714 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 274743us
I20251025 20:04:15.259302 31958 raft_consensus.cc:2749] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251025 20:04:15.257187 31953 consensus_queue.cc:237] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.259001 31801 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.052484 31885 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:15.058228 31943 log.cc:927] Time spent T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.264s user 0.000s sys 0.001s
W20251025 20:04:15.072459 31960 log.cc:927] Time spent T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.211s user 0.000s sys 0.002s
I20251025 20:04:15.102097 31679 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:15.262176 31885 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.262724 31679 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.264158 30719 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:15.102849 31895 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:15.264587 31885 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.117379 31932 raft_consensus.cc:2749] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:15.125218 30673 raft_consensus.cc:1275] T e4bb9e40b009470eb783f9bf63e0c3ba P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251025 20:04:15.273041 31809 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.274065 31610 consensus_queue.cc:1048] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:04:15.132683 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:15.278267 30671 raft_consensus.cc:3060] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:15.132186 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:15.146327 31871 consensus_meta.cc:287] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.735s user 0.010s sys 0.004s
W20251025 20:04:15.122923 31927 log.cc:927] Time spent T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.507s user 0.004s sys 0.007s
I20251025 20:04:15.148347 30677 raft_consensus.cc:1275] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20251025 20:04:15.281424 30677 pending_rounds.cc:85] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 2
I20251025 20:04:15.281714 30677 pending_rounds.cc:107] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251025 20:04:15.282006 30677 raft_consensus.cc:2889] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251025 20:04:15.133710 31955 log.cc:927] Time spent T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20251025 20:04:15.122521 31913 log.cc:927] Time spent T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.616s user 0.009s sys 0.007s
W20251025 20:04:15.179917 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:15.284739 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.181806 31940 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.157096 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:15.182886 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.183012 31855 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:15.180415 31769 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:15.183972 31968 log.cc:927] Time spent T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20251025 20:04:15.148690 31963 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:14.934799 30749 rpcz_store.cc:276] 1025 20:04:13.528312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:13.538191 (+ 9879us) service_pool.cc:225] Handling call
1025 20:04:13.640453 (+102262us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:13.794358 (+153905us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:13.801726 (+ 7368us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:13.880649 (+ 78923us) log.cc:844] Serialized 26 byte log entry
1025 20:04:14.334257 (+453608us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:14.395924 (+ 61667us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:14.582485 (+186561us) spinlock_profiling.cc:243] Waited 66.7 ms on lock 0x7b540019cb54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:14.582544 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:14.691717 (+109173us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:14.846447 (+154730us) raft_consensus.cc:1685] finished
1025 20:04:14.849158 (+ 2711us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:14.862648 (+ 13490us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":41877,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":7190,"spinlock_wait_cycles":187369728,"thread_start_us":399040,"threads_started":1,"wal-append.queue_time_us":573715}
W20251025 20:04:15.194393 31491 consensus_meta.cc:287] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 1.021s user 0.014s sys 0.000s
I20251025 20:04:15.296701 30671 raft_consensus.cc:2468] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:15.147749 31812 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:15.230845 31478 consensus_meta.cc:287] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.579s user 0.003s sys 0.010s
I20251025 20:04:15.231225 31567 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:15.193409 31751 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.232847 31732 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:14.982769 30567 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:15.255558 31945 consensus_queue.cc:1059] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:04:14.916294 31696 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.054704 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:14.993546 31930 log.cc:927] Time spent T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.422s user 0.001s sys 0.001s
I20251025 20:04:15.259058 30645 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:15.036764 31946 raft_consensus.cc:2804] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election won for term 4
W20251025 20:04:15.334719 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 352044us
I20251025 20:04:15.336097 30567 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:15.039285 31947 raft_consensus.cc:2749] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251025 20:04:15.109835 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.261555 31801 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.269781 31941 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:15.265043 31895 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.272743 31958 raft_consensus.cc:2749] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:15.264717 31679 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.279407 30719 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:15.299252 30746 raft_consensus.cc:1240] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:15.299194 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:15.302879 30716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.311998 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.326357 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.332947 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:15.345332 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.341029 30567 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:15.351943 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 92950.9us
I20251025 20:04:15.352558 30670 raft_consensus.cc:1240] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:15.353130 30676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 442) took 1047 ms. Trace:
I20251025 20:04:15.409045 30676 rpcz_store.cc:276] 1025 20:04:14.305276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:14.331471 (+ 26195us) service_pool.cc:225] Handling call
1025 20:04:14.331900 (+ 429us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:14.332500 (+ 600us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:14.332567 (+ 67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:14.412243 (+ 79676us) log.cc:844] Serialized 26 byte log entry
1025 20:04:14.833206 (+420963us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:14.833268 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:14.833372 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:15.352923 (+519551us) raft_consensus.cc:1685] finished
1025 20:04:15.352943 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:15.353066 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":413,"raft.queue_time_us":2686,"raft.run_cpu_time_us":548,"raft.run_wall_time_us":111550,"thread_start_us":362850,"threads_started":1,"wal-append.queue_time_us":517115}
W20251025 20:04:15.411012 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 131702us
W20251025 20:04:15.356698 31920 log.cc:927] Time spent T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.850s user 0.004s sys 0.007s
I20251025 20:04:15.317593 31567 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.296422 31855 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.354594 31932 consensus_queue.cc:1048] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.016s
I20251025 20:04:15.379148 31801 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:15.300101 31491 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.377056 31696 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:15.364823 31979 log.cc:927] Time spent T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20251025 20:04:15.308539 31478 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.340745 31940 raft_consensus.cc:2749] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:15.340186 31948 raft_consensus.cc:2749] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:15.357368 31941 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.378865 31751 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:15.379148 31975 log.cc:927] Time spent T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.103s user 0.000s sys 0.002s
I20251025 20:04:15.327366 31732 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.376660 31679 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:15.385557 31871 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:15.383147 31812 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:15.450090 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.450594 31567 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.452517 31732 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.452966 31679 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.393726 30570 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:15.353461 31885 raft_consensus.cc:2749] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:15.378865 31895 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.378329 31947 consensus_queue.cc:1059] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:15.363628 31946 raft_consensus.cc:697] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:15.407606 30674 raft_consensus.cc:1240] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:15.459790 31679 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.296326 31963 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:15.412564 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 71629.8us
I20251025 20:04:15.413210 30645 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:15.406577 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.416764 31801 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.415246 31794 raft_consensus.cc:2804] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:15.408650 31965 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:15.407056 31809 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:15.434012 30679 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:15.435654 30675 raft_consensus.cc:3055] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:15.412819 31847 raft_consensus.cc:2749] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:15.439361 30673 raft_consensus.cc:1275] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:15.440312 30719 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:15.442302 30752 raft_consensus.cc:1240] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:15.449378 31491 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:15.449368 30747 raft_consensus.cc:1240] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: []
W20251025 20:04:15.455593 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 61990.3us
I20251025 20:04:15.460326 31946 consensus_queue.cc:237] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.483028 31941 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:15.490788 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 50535.6us
I20251025 20:04:15.491108 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:15.492372 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.493145 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.493516 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.494320 30745 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:15.494459 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:04:15.462015 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.494885 31491 raft_consensus.cc:2749] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:15.463109 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:15.464497 31849 consensus_queue.cc:1059] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251025 20:04:15.470778 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 62230.1us
I20251025 20:04:15.494396 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.498476 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:15.436272 31969 consensus_queue.cc:1048] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.032s
I20251025 20:04:15.472105 31794 raft_consensus.cc:697] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:15.475427 31963 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:15.468050 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.478107 31801 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.478263 31855 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.478752 31942 raft_consensus.cc:2749] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:15.479189 31478 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:15.479290 31414 raft_consensus.cc:3055] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:15.441566 31945 consensus_queue.cc:1048] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251025 20:04:15.456875 31732 raft_consensus.cc:2749] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:15.481115 30675 raft_consensus.cc:740] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:15.461086 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:15.353976 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.511320 30679 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:15.516571 30675 consensus_queue.cc:260] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.518545 30645 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:15.497262 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.523299 30570 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:15.513223 30567 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:15.525244 30675 raft_consensus.cc:3060] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:15.456375 31871 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.512825 31414 raft_consensus.cc:740] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:15.512521 31567 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:15.529870 31855 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:15.484238 31965 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.539104 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.536278 31947 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:15.525440 31935 consensus_queue.cc:1059] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20251025 20:04:15.563777 31567 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.540830 31794 consensus_queue.cc:237] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.530030 31801 consensus_queue.cc:1048] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.029s
I20251025 20:04:15.542521 31942 raft_consensus.cc:2749] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:15.542652 31847 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:15.554549 31850 consensus_queue.cc:1048] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20251025 20:04:15.515776 31958 consensus_queue.cc:1059] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20251025 20:04:15.556022 31679 raft_consensus.cc:2804] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:15.541384 31849 consensus_queue.cc:1059] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
W20251025 20:04:15.587028 31988 log.cc:927] Time spent T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20251025 20:04:15.558988 31855 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:15.587828 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 76225.5us
I20251025 20:04:15.586316 31847 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:15.594396 31679 raft_consensus.cc:697] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:15.600499 31679 consensus_queue.cc:237] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.601066 31855 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.568819 31940 consensus_queue.cc:1048] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
W20251025 20:04:15.603044 31985 log.cc:927] Time spent T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20251025 20:04:15.603335 31965 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.604660 30744 raft_consensus.cc:1240] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:15.609043 30675 raft_consensus.cc:1275] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:15.617306 30719 leader_election.cc:304] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:15.576926 31991 consensus_queue.cc:1048] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20251025 20:04:15.598757 31847 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:15.587067 31947 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.597255 31935 raft_consensus.cc:3055] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:15.636485 30597 raft_consensus.cc:1240] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251025 20:04:15.638291 31990 raft_consensus.cc:2749] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:04:15.639406 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.587790 31414 consensus_queue.cc:260] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.642858 30598 raft_consensus.cc:1240] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:15.644268 30645 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:15.616273 31850 consensus_queue.cc:1059] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.028s
I20251025 20:04:15.650519 30595 raft_consensus.cc:1275] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:15.650859 31850 raft_consensus.cc:3055] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:15.651312 31850 raft_consensus.cc:740] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:15.652323 31936 consensus_queue.cc:1048] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
W20251025 20:04:15.652837 31981 log.cc:927] Time spent T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20251025 20:04:15.631232 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.653484 30744 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:15.656309 31892 raft_consensus.cc:3055] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:15.656751 31892 raft_consensus.cc:740] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
@ 0x7fd81ec7ee30 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
I20251025 20:04:15.657688 31892 consensus_queue.cc:260] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.659185 31892 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:15.660184 30676 raft_consensus.cc:1240] T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:15.652271 31850 consensus_queue.cc:260] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.651222 30645 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:15.666762 30570 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:15.665704 31940 raft_consensus.cc:2749] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 20:04:15.644320 30597 raft_consensus.cc:1240] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:15.616185 31965 consensus_queue.cc:1059] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20251025 20:04:15.674218 31769 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:15.674643 31769 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:15.674000 31990 consensus_queue.cc:1059] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:04:15.675024 31769 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:15.636762 31801 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:15.677660 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:15.632413 31935 raft_consensus.cc:740] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:15.723304 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 72181.2us
I20251025 20:04:15.765009 30598 raft_consensus.cc:1240] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:15.771227 30595 raft_consensus.cc:1275] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:15.771739 30675 raft_consensus.cc:1275] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:15.668859 31942 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:15.772392 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.720345 31941 raft_consensus.cc:2749] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:15.772940 31567 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:15.763257 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.763204 31414 raft_consensus.cc:3060] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:15.777860 30601 raft_consensus.cc:1240] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:15.801756 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:15.803247 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:15.804127 30595 raft_consensus.cc:2368] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 4.
I20251025 20:04:15.804983 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:15.806306 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:15.819250 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.723279 31855 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:15.807092 30750 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:15.762400 31850 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:15.832311 31935 consensus_queue.cc:260] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:15.833279 31994 log.cc:927] Time spent T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20251025 20:04:15.834054 31567 raft_consensus.cc:493] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:15.869145 31567 raft_consensus.cc:515] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.870460 31953 raft_consensus.cc:3055] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:15.870931 31953 raft_consensus.cc:740] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:15.871147 30671 raft_consensus.cc:1240] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:15.856314 31969 consensus_queue.cc:1048] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20251025 20:04:15.836138 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:15.783125 30672 raft_consensus.cc:3055] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Stepping down as leader of term 1
W20251025 20:04:15.786898 31989 log.cc:927] Time spent T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
I20251025 20:04:15.841071 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.827618 31942 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.843863 31610 raft_consensus.cc:3055] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:15.850644 31941 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:15.851678 30645 connection.cc:232] RPC call timeout handler was delayed by -4.1856580159999339s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
I20251025 20:04:15.860296 31992 consensus_queue.cc:1059] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 20:04:15.865326 31809 raft_consensus.cc:3055] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
W20251025 20:04:15.865974 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.871950 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:15.854463 30719 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:15.856540 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:15.854408 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:15.906075 30601 raft_consensus.cc:2368] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 4.
W20251025 20:04:15.877154 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.849105 30677 raft_consensus.cc:1275] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:15.871942 31953 consensus_queue.cc:260] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.829312 31751 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:15.878515 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 333152us
I20251025 20:04:15.879984 30670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 713) took 1964 ms. Trace:
I20251025 20:04:15.884162 30750 raft_consensus.cc:1240] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:15.886018 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:15.790733 31895 raft_consensus.cc:3055] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:15.896587 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.906093 30597 raft_consensus.cc:2410] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251025 20:04:15.897267 30746 raft_consensus.cc:1240] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20251025 20:04:15.795220 31871 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:15.898764 30645 connection.cc:232] RPC call timeout handler was delayed by -4.09835801600002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19840000000000002s for a response
W20251025 20:04:15.918516 30645 connection.cc:232] RPC call timeout handler was delayed by -3.9840580160000627s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
I20251025 20:04:15.919780 30645 leader_election.cc:304] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:15.921223 30645 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:15.922134 31940 raft_consensus.cc:2804] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:15.922258 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.922555 31940 raft_consensus.cc:493] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:15.922941 31940 raft_consensus.cc:3060] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:15.923539 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.924582 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.926252 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.926100 31696 consensus_queue.cc:1048] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:15.927609 31696 raft_consensus.cc:2749] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251025 20:04:15.928017 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.723409 31801 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.928730 30677 raft_consensus.cc:2393] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:15.929502 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:15.930141 31801 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.900635 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.901814 30748 raft_consensus.cc:1240] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:15.908202 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:15.931845 30598 raft_consensus.cc:2393] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:15.932848 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:15.871197 31567 leader_election.cc:290] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.908826 30670 rpcz_store.cc:276] 1025 20:04:13.916106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:13.920189 (+ 4083us) service_pool.cc:225] Handling call
1025 20:04:13.969693 (+ 49504us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:15.408716 (+1439023us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:15.879690 (+470974us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540017acd4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:15.879917 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3173075712}
I20251025 20:04:15.878243 30672 raft_consensus.cc:740] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:15.914734 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.915582 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.935359 30672 consensus_queue.cc:260] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.936645 30672 raft_consensus.cc:3060] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:15.942535 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:15.942538 30598 raft_consensus.cc:2410] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:15.948171 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:15.950661 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.952051 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.953258 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.949807 31940 raft_consensus.cc:515] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.953886 30670 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:15.954237 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.955354 30677 raft_consensus.cc:2468] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:15.957180 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.927289 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.958608 30679 raft_consensus.cc:2468] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:15.960024 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:15.962942 31940 leader_election.cc:290] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.880146 31855 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:15.974056 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:15.974665 31871 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:15.974838 30567 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:15.953852 30672 raft_consensus.cc:1275] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:15.975237 31855 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.894323 31990 consensus_queue.cc:1048] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
I20251025 20:04:15.976881 31947 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.979597 31965 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:15.979969 31947 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:15.980103 31965 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.980553 31947 raft_consensus.cc:697] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 5 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:15.982132 31801 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:15.982650 31801 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.984875 31801 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.987634 31965 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.988565 31935 raft_consensus.cc:3060] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:15.988921 31965 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:15.900256 31809 raft_consensus.cc:740] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:15.989941 31965 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.993189 31809 consensus_queue.cc:260] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.994740 31809 raft_consensus.cc:3060] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:15.998198 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:15.998862 31953 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:15.916949 31895 raft_consensus.cc:740] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:15.975059 31945 raft_consensus.cc:3055] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:15.893661 31610 raft_consensus.cc:740] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:15.910569 31849 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:16.014360 31942 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:16.015524 31491 raft_consensus.cc:2749] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:16.042205 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:16.018537 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20251025 20:04:16.018832 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 164453us
I20251025 20:04:16.034957 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:16.048964 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:16.051723 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:16.056221 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.062341 30679 raft_consensus.cc:1275] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:16.068794 31895 consensus_queue.cc:260] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.070801 31895 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:16.079546 30598 raft_consensus.cc:2468] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:04:16.081255 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:16.082526 30598 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:15.909152 31941 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.022428 31945 raft_consensus.cc:740] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:16.091777 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.105005 30672 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:15.994753 31965 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:15.981468 31947 consensus_queue.cc:237] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.003669 31751 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:15.975237 31871 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.043833 31849 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.048746 31936 consensus_queue.cc:1059] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
W20251025 20:04:16.116570 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.137563 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.150760 30673 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:16.150919 30675 raft_consensus.cc:1240] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:16.136005 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.145201 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.146397 30597 raft_consensus.cc:2368] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:16.156973 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:16.148241 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.158859 30753 raft_consensus.cc:2393] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:16.163636 30597 raft_consensus.cc:2393] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:16.141187 30602 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:16.132830 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:16.164103 31847 consensus_meta.cc:287] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.514s user 0.008s sys 0.004s
I20251025 20:04:16.137187 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:16.200310 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.201164 30750 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:16.188695 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.202507 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.203368 30679 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:16.181846 30556 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Sending DeleteTablet for 3 replicas of tablet e8c971a6d4854180b4d243fa79fbbc11
I20251025 20:04:16.191772 30678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 459) took 1394 ms. Trace:
W20251025 20:04:16.208705 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 287522us
I20251025 20:04:16.182410 30598 raft_consensus.cc:2468] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:04:16.221997 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:16.230304 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 49270.6us
I20251025 20:04:16.233291 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.160320 31788 consensus_queue.cc:1048] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.046s
I20251025 20:04:16.182819 31626 consensus_queue.cc:1059] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20251025 20:04:16.262367 30603 raft_consensus.cc:1240] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:16.269546 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.250559 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.250687 30645 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.278385 30567 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.235749 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:16.282229 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.237370 30570 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:16.262117 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:16.199082 31932 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:16.288698 30752 raft_consensus.cc:1217] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
W20251025 20:04:16.289273 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 58113.4us
I20251025 20:04:16.320264 30752 raft_consensus.cc:1240] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:16.320551 30719 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:16.236440 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:16.321091 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 83811.7us
W20251025 20:04:16.320920 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 70241us
W20251025 20:04:16.244167 32000 log.cc:927] Time spent T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
W20251025 20:04:16.244355 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.274147 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.322703 30745 raft_consensus.cc:1275] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251025 20:04:16.323935 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 306045us
I20251025 20:04:16.316745 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.331878 30746 raft_consensus.cc:1275] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20251025 20:04:16.330633 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 52397.4us
W20251025 20:04:16.290059 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.334637 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.333977 30567 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:16.294256 31769 consensus_meta.cc:287] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.619s user 0.004s sys 0.010s
I20251025 20:04:16.337049 30567 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:16.383198 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 55940.3us
I20251025 20:04:16.388897 30645 leader_election.cc:400] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 4.
I20251025 20:04:16.244881 31948 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:16.389643 30645 leader_election.cc:403] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251025 20:04:16.390425 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:16.390583 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:16.393684 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.393728 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:16.393976 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:16.394488 30748 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:16.397001 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:16.397708 30597 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:16.322983 31940 raft_consensus.cc:2749] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:16.330091 31679 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:16.401974 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.205915 31962 consensus_queue.cc:1059] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.184s
I20251025 20:04:16.402890 30645 leader_election.cc:400] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 4.
I20251025 20:04:16.403329 30556 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Sending DeleteTablet for 3 replicas of tablet fea0c74c9c0c4a69afe072c45c4cbe99
I20251025 20:04:16.203502 31847 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.334830 31567 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 20:04:16.400240 31955 log.cc:927] Time spent T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20251025 20:04:16.414835 30597 raft_consensus.cc:1240] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:16.426553 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.427809 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:16.388075 31997 raft_consensus.cc:2749] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:16.411664 30645 leader_election.cc:403] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251025 20:04:16.332863 31932 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.414042 30598 raft_consensus.cc:1240] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:16.436872 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.464066 30746 raft_consensus.cc:2410] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:16.388672 32013 raft_consensus.cc:2749] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:16.400655 31942 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:16.393263 31732 raft_consensus.cc:2749] T 06d7fa7a698c4890af235aed97fd6fb5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:16.437173 30602 raft_consensus.cc:2468] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 1.
I20251025 20:04:16.436757 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.466434 30556 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Sending DeleteTablet for 3 replicas of tablet da8a0eaa68044835a4cf94340ca25ac5
I20251025 20:04:16.465022 31940 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:16.469772 31847 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:16.406816 31801 consensus_queue.cc:1048] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20251025 20:04:16.470490 30601 raft_consensus.cc:2368] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 6.
I20251025 20:04:16.510286 30586 tablet_service.cc:1552] Processing DeleteTablet for tablet e8c971a6d4854180b4d243fa79fbbc11 with delete_type TABLET_DATA_DELETED (Replaced by f69f57fe417f492d9b82ed651b9d70c1 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:34862
W20251025 20:04:16.515386 31809 consensus_meta.cc:287] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.520s user 0.007s sys 0.004s
W20251025 20:04:16.517710 30719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.520536 31876 raft_consensus.cc:3055] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:16.521121 31876 raft_consensus.cc:740] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:16.521996 31932 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:16.522409 30738 tablet_service.cc:1552] Processing DeleteTablet for tablet e8c971a6d4854180b4d243fa79fbbc11 with delete_type TABLET_DATA_DELETED (Replaced by f69f57fe417f492d9b82ed651b9d70c1 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:53746
I20251025 20:04:16.522091 31876 consensus_queue.cc:260] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.523999 30734 tablet_service.cc:1552] Processing DeleteTablet for tablet fea0c74c9c0c4a69afe072c45c4cbe99 with delete_type TABLET_DATA_DELETED (Replaced by 7294f698d29e4c919745b460e4e25eeb at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:53746
I20251025 20:04:16.525445 30556 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Sending DeleteTablet for 3 replicas of tablet 8e7a1b97c4aa4812a26b178e32476392
I20251025 20:04:16.478169 31849 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:16.526404 30590 tablet_service.cc:1552] Processing DeleteTablet for tablet da8a0eaa68044835a4cf94340ca25ac5 with delete_type TABLET_DATA_DELETED (Replaced by 01d56eb384124dfcb1bcd35d77e89f0e at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:34862
I20251025 20:04:16.527017 31871 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:16.527374 31876 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:16.527923 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.529254 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.524076 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.530848 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:16.531109 30746 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:16.535356 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:16.470666 30587 tablet_service.cc:1552] Processing DeleteTablet for tablet fea0c74c9c0c4a69afe072c45c4cbe99 with delete_type TABLET_DATA_DELETED (Replaced by 7294f698d29e4c919745b460e4e25eeb at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:34862
W20251025 20:04:16.471470 32009 log.cc:927] Time spent T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
W20251025 20:04:16.471843 32003 log.cc:927] Time spent T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.289s user 0.001s sys 0.000s
I20251025 20:04:16.413540 31934 consensus_queue.cc:1059] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20251025 20:04:16.427726 31941 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:16.472527 32006 log.cc:927] Time spent T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Append to log took a long time: real 0.222s user 0.000s sys 0.000s
I20251025 20:04:16.414469 31812 consensus_queue.cc:1048] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
W20251025 20:04:16.473476 32007 log.cc:927] Time spent T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
I20251025 20:04:16.338270 31754 consensus_queue.cc:1059] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
I20251025 20:04:16.404374 31679 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:16.476562 30598 raft_consensus.cc:1240] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3. Ops: [2.1-2.1]
W20251025 20:04:16.476744 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 46292.9us
W20251025 20:04:16.477535 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 140570us
I20251025 20:04:16.388345 31769 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.393266 31970 consensus_queue.cc:1048] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20251025 20:04:16.478238 31855 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
W20251025 20:04:16.469071 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 66307us
I20251025 20:04:16.415783 31969 consensus_queue.cc:1048] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20251025 20:04:16.474812 31751 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:16.407228 31948 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:16.533658 31892 consensus_meta.cc:287] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.872s user 0.013s sys 0.000s
I20251025 20:04:16.545605 30750 raft_consensus.cc:1240] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251025 20:04:16.545490 30567 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:16.549947 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 147457us
I20251025 20:04:16.608505 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c971a6d4854180b4d243fa79fbbc11" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:16.551956 30670 raft_consensus.cc:1240] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:16.552453 30598 raft_consensus.cc:1240] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251025 20:04:16.582476 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.590344 30645 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:04:16.543573 31999 log.cc:927] Time spent T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.346s user 0.001s sys 0.001s
I20251025 20:04:16.601698 30677 raft_consensus.cc:1275] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:16.604926 30556 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Sending DeleteTablet for 3 replicas of tablet 3f1269e0f1364a69b9a36ce877b0da5e
I20251025 20:04:16.611606 30645 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:16.613670 30595 raft_consensus.cc:2368] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 6.
W20251025 20:04:16.580564 32004 log.cc:927] Time spent T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
W20251025 20:04:16.555239 31850 consensus_meta.cc:287] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.688s user 0.010s sys 0.003s
I20251025 20:04:16.552886 31679 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:16.619450 30598 raft_consensus.cc:1240] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:16.622695 30645 leader_election.cc:400] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:16.623072 30672 raft_consensus.cc:3055] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:16.623389 30672 raft_consensus.cc:740] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:16.623942 30672 consensus_queue.cc:260] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.624595 30672 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:16.620772 30734 tablet_service.cc:1552] Processing DeleteTablet for tablet da8a0eaa68044835a4cf94340ca25ac5 with delete_type TABLET_DATA_DELETED (Replaced by 01d56eb384124dfcb1bcd35d77e89f0e at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:53746
I20251025 20:04:16.625496 30587 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e7a1b97c4aa4812a26b178e32476392 with delete_type TABLET_DATA_DELETED (Replaced by d8afaad4899545a9a4169026b63c0699 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:34862
I20251025 20:04:16.599133 31847 raft_consensus.cc:3055] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:16.629964 30556 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Sending DeleteTablet for 3 replicas of tablet 4614bdb7f750431fa840cabccf782b4b
I20251025 20:04:16.632542 30570 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:16.538928 31940 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.642421 30598 raft_consensus.cc:1275] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251025 20:04:16.658200 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 112834us
I20251025 20:04:16.659139 30567 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.630950 31847 raft_consensus.cc:740] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:16.648505 30587 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f1269e0f1364a69b9a36ce877b0da5e with delete_type TABLET_DATA_DELETED (Replaced by 6164729145ad47d6bbc2e7b3c6b49a22 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:34862
I20251025 20:04:16.659814 31895 raft_consensus.cc:2749] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:16.667330 31940 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:16.668483 32010 log.cc:927] Time spent T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20251025 20:04:16.669243 30747 raft_consensus.cc:2468] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:04:16.669950 31769 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:16.670068 30604 raft_consensus.cc:2468] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:16.672065 30674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 765) took 1014 ms. Trace:
I20251025 20:04:16.672513 30674 rpcz_store.cc:276] 1025 20:04:15.657489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:15.661831 (+ 4342us) service_pool.cc:225] Handling call
1025 20:04:15.674686 (+ 12855us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:15.675060 (+ 374us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:15.675092 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:15.675647 (+ 555us) log.cc:844] Serialized 26 byte log entry
1025 20:04:16.086475 (+410828us) raft_consensus.cc:1645] Marking committed up to 0
1025 20:04:16.086506 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:16.086558 (+ 52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:16.671766 (+585208us) raft_consensus.cc:1685] finished
1025 20:04:16.671834 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:16.671988 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1084,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":33729,"thread_start_us":410458,"threads_started":1,"wal-append.queue_time_us":596985}
I20251025 20:04:16.630018 31941 raft_consensus.cc:2804] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:16.674168 31941 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:16.674559 31941 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:16.679111 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 46647.3us
I20251025 20:04:16.679616 30567 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.683041 30556 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Sending DeleteTablet for 3 replicas of tablet 0d9831caf5d44fc080590d41bd68a083
I20251025 20:04:16.684317 30672 raft_consensus.cc:1275] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251025 20:04:16.684803 30672 pending_rounds.cc:85] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 2
W20251025 20:04:16.684808 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.685504 30672 pending_rounds.cc:107] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251025 20:04:16.686693 30672 raft_consensus.cc:2889] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:04:16.689826 30567 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.691680 30567 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:16.693287 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 70733.9us
I20251025 20:04:16.693595 30602 raft_consensus.cc:1217] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20251025 20:04:16.692961 30567 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.694182 30602 raft_consensus.cc:1240] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 1. Current term is 2. Ops: [1.1-1.1]
W20251025 20:04:16.683920 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.694347 30645 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:16.700196 32025 tablet_replica.cc:333] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: stopping tablet replica
I20251025 20:04:16.691037 32021 raft_consensus.cc:2804] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:16.701027 32021 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:16.701447 32021 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:16.702121 31895 raft_consensus.cc:2749] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:16.704315 32011 raft_consensus.cc:2804] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:04:16.705762 32023 tablet_replica.cc:333] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: stopping tablet replica
W20251025 20:04:16.707304 32008 log.cc:927] Time spent T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20251025 20:04:16.709610 30586 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d9831caf5d44fc080590d41bd68a083 with delete_type TABLET_DATA_DELETED (Replaced by 5ee1ba6e04b74e1faf5c9819144cd740 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:34862
I20251025 20:04:16.691201 31751 raft_consensus.cc:2749] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251025 20:04:16.719285 30587 tablet_service.cc:1552] Processing DeleteTablet for tablet 4614bdb7f750431fa840cabccf782b4b with delete_type TABLET_DATA_DELETED (Replaced by 8360e93e9b6d471791c2bd59bf8292ab at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:34862
I20251025 20:04:16.719569 31847 consensus_queue.cc:260] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.722123 31847 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:16.701197 31945 consensus_queue.cc:260] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:16.732674 31935 consensus_meta.cc:287] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.744s user 0.015s sys 0.000s
I20251025 20:04:16.733400 31945 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:16.718230 30556 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Sending DeleteTablet for 3 replicas of tablet 580e75e8dcd949e39aff92ada50b40be
I20251025 20:04:16.697741 31769 raft_consensus.cc:2749] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:16.738829 31940 raft_consensus.cc:2804] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election won for term 4
W20251025 20:04:16.740404 32001 log.cc:927] Time spent T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.445s user 0.005s sys 0.000s
I20251025 20:04:16.740447 30587 tablet_service.cc:1552] Processing DeleteTablet for tablet 580e75e8dcd949e39aff92ada50b40be with delete_type TABLET_DATA_DELETED (Replaced by 2d6dd59f914c41758a18803991215caf at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:34862
I20251025 20:04:16.741990 30600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 748) took 1108 ms. Trace:
I20251025 20:04:16.220774 30678 rpcz_store.cc:276] 1025 20:04:14.797526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:14.820287 (+ 22761us) service_pool.cc:225] Handling call
1025 20:04:14.844235 (+ 23948us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:14.844495 (+ 260us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:15.798226 (+953731us) spinlock_profiling.cc:243] Waited 39.7 ms on lock 0x7b1800037e40. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8278d09a1 00007fd8278dd371 00007fd8216e255c 00007fd81ea7097b 00007fd827849219 00007fd81ea8cc4b 00007fd81ea8aeef 00007fd81ea8aa05 00007fd825529b27 00007fd821b44537 00007fd821b53c4f
1025 20:04:15.798399 (+ 173us) log.cc:844] Serialized 14 byte log entry
1025 20:04:16.191328 (+392929us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:16.191403 (+ 75us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:16.191463 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:16.191557 (+ 94us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:16.191699 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65577,"prepare.queue_time_us":220125,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":102,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":15214,"spinlock_wait_cycles":91309312,"thread_start_us":553834,"threads_started":2}
I20251025 20:04:16.742432 30600 rpcz_store.cc:276] 1025 20:04:15.633786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:15.634035 (+ 249us) service_pool.cc:225] Handling call
1025 20:04:15.635104 (+ 1069us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:15.643368 (+ 8264us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:15.643406 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:15.643900 (+ 494us) log.cc:844] Serialized 26 byte log entry
1025 20:04:16.073114 (+429214us) raft_consensus.cc:1645] Marking committed up to 3
1025 20:04:16.073164 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:16.073241 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:16.739490 (+666249us) spinlock_profiling.cc:243] Waited 21.5 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:16.741633 (+ 2143us) raft_consensus.cc:1685] finished
1025 20:04:16.741663 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:16.741912 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1984,"raft.queue_time_us":4807,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"spinlock_wait_cycles":49471360,"thread_start_us":417500,"threads_started":1,"wal-append.queue_time_us":546924}
I20251025 20:04:16.745203 30587 tablet_service.cc:1505] Processing CreateTablet for tablet f69f57fe417f492d9b82ed651b9d70c1 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251025 20:04:16.729240 30595 raft_consensus.cc:1240] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 5 LEADER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:16.749545 30595 raft_consensus.cc:1240] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20251025 20:04:16.695278 31953 consensus_meta.cc:287] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.696s user 0.015s sys 0.000s
I20251025 20:04:16.684711 31754 raft_consensus.cc:3060] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:16.751703 30595 raft_consensus.cc:1275] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:16.753621 31610 consensus_queue.cc:260] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
W20251025 20:04:16.754681 32012 log.cc:927] Time spent T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.259s user 0.001s sys 0.000s
I20251025 20:04:16.754832 31610 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:16.756915 31948 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:16.695865 31947 raft_consensus.cc:2749] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251025 20:04:16.761401 30716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.762225 30596 raft_consensus.cc:1275] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:16.764746 30604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 764) took 1043 ms. Trace:
I20251025 20:04:16.765141 30604 rpcz_store.cc:276] 1025 20:04:15.721494 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:15.885737 (+164243us) service_pool.cc:225] Handling call
1025 20:04:16.764582 (+878845us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54001c16d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:16.764686 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1654273408}
I20251025 20:04:16.766634 30604 raft_consensus.cc:1240] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:16.767920 30604 raft_consensus.cc:1275] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:16.770416 31946 raft_consensus.cc:3055] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:16.770815 31946 raft_consensus.cc:740] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:16.771677 31946 consensus_queue.cc:260] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.773186 31946 raft_consensus.cc:3060] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:16.775931 30587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69f57fe417f492d9b82ed651b9d70c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:16.781370 30603 raft_consensus.cc:1240] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 6. Ops: [4.2-4.2]
W20251025 20:04:16.783946 32005 log.cc:927] Time spent T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.375s user 0.000s sys 0.001s
I20251025 20:04:16.763671 30596 raft_consensus.cc:1275] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:16.785178 30676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 766) took 1127 ms. Trace:
I20251025 20:04:16.785543 30676 rpcz_store.cc:276] 1025 20:04:15.657785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:15.662621 (+ 4836us) service_pool.cc:225] Handling call
1025 20:04:15.662952 (+ 331us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:15.663337 (+ 385us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:15.663371 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:15.663899 (+ 528us) log.cc:844] Serialized 26 byte log entry
1025 20:04:16.142860 (+478961us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:16.142943 (+ 83us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:16.143049 (+ 106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:16.783179 (+640130us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:16.784857 (+ 1678us) raft_consensus.cc:1685] finished
1025 20:04:16.784884 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:16.785111 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1104,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9977,"spinlock_wait_cycles":8439040,"thread_start_us":477135,"threads_started":1,"wal-append.queue_time_us":725326}
W20251025 20:04:16.792994 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:16.794598 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 187330us
W20251025 20:04:16.797574 30719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.798712 30567 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.799695 31948 raft_consensus.cc:2804] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251025 20:04:16.799736 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 100909us
I20251025 20:04:16.800228 31948 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:16.800642 31948 raft_consensus.cc:3060] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:16.801106 30567 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.802481 31962 consensus_queue.cc:1059] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251025 20:04:16.802774 31947 raft_consensus.cc:2804] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:16.803282 31947 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:16.803771 31947 raft_consensus.cc:3060] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:16.804658 31744 raft_consensus.cc:3055] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:16.805146 31744 raft_consensus.cc:740] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:16.806147 31744 consensus_queue.cc:260] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.803612 31962 consensus_queue.cc:1048] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:04:16.807530 31744 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:16.809554 31941 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
@ 0x7fd81ec5ca32 testing::Test::Run() at ??:0
I20251025 20:04:16.812325 30603 raft_consensus.cc:1275] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251025 20:04:16.824870 31679 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.819139 31942 raft_consensus.cc:2749] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 4.
I20251025 20:04:16.699460 30645 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:04:16.823145 30643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.713794 31855 raft_consensus.cc:2749] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 4.
I20251025 20:04:16.826498 30598 raft_consensus.cc:1275] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:16.824894 31940 raft_consensus.cc:697] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:16.846352 30672 raft_consensus.cc:3055] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:16.846750 30672 raft_consensus.cc:740] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:16.847482 30672 consensus_queue.cc:260] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.840703 30567 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.814338 30716 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:16.843672 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.852948 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:16.853711 30750 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:16.854278 30675 raft_consensus.cc:1240] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:16.858031 30567 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.858953 30672 raft_consensus.cc:3060] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
W20251025 20:04:16.850708 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.862815 31769 raft_consensus.cc:2749] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:16.863170 30679 raft_consensus.cc:1275] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:16.863891 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.863029 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:16.865707 30745 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:16.865674 30500 catalog_manager.cc:5649] T 3a03f8a32da8499dbc0e2a426d214001 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:16.867498 30500 catalog_manager.cc:5649] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 2, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:16.870631 30744 raft_consensus.cc:1275] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:04:16.871512 31855 raft_consensus.cc:2804] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:16.872002 31855 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:16.871819 30586 tablet_service.cc:1505] Processing CreateTablet for tablet 7294f698d29e4c919745b460e4e25eeb (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251025 20:04:16.872404 31855 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:16.873158 30738 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e7a1b97c4aa4812a26b178e32476392 with delete_type TABLET_DATA_DELETED (Replaced by d8afaad4899545a9a4169026b63c0699 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:53746
I20251025 20:04:16.873684 31769 raft_consensus.cc:2749] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:16.873342 30586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7294f698d29e4c919745b460e4e25eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:16.876876 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.878471 31940 consensus_queue.cc:237] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:16.880115 31941 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:16.871726 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:16.881254 31855 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.881541 31751 consensus_queue.cc:1048] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251025 20:04:16.888309 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 74080.9us
I20251025 20:04:16.901607 30671 raft_consensus.cc:1240] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:16.869625 30500 catalog_manager.cc:5649] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 2 to 3, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:16.905928 30567 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.907207 30500 catalog_manager.cc:5649] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 2, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
W20251025 20:04:16.908171 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 208826us
I20251025 20:04:16.885794 31855 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:16.910913 30567 leader_election.cc:304] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.852800 32011 raft_consensus.cc:697] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:16.910614 30645 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:16.917719 30672 raft_consensus.cc:1275] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251025 20:04:16.925016 30567 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.925745 30734 tablet_service.cc:1552] Processing DeleteTablet for tablet 4614bdb7f750431fa840cabccf782b4b with delete_type TABLET_DATA_DELETED (Replaced by 8360e93e9b6d471791c2bd59bf8292ab at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:53746
I20251025 20:04:16.929630 30672 pending_rounds.cc:85] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 2
I20251025 20:04:16.930857 30672 pending_rounds.cc:107] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251025 20:04:16.931195 30672 raft_consensus.cc:2889] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:04:16.932092 30567 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.927590 32025 raft_consensus.cc:2243] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:04:16.908952 30500 catalog_manager.cc:5649] T 1df036864f934f2d81add520300e9ed6 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 2 to 3, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:16.935102 30645 leader_election.cc:304] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:16.940240 30567 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:16.941380 30645 leader_election.cc:400] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 6.
I20251025 20:04:16.941562 30673 raft_consensus.cc:1240] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:16.941877 30645 leader_election.cc:403] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:04:16.923625 31895 raft_consensus.cc:2804] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:16.946272 30500 catalog_manager.cc:5649] T 5f27e13dcf894a1487ed057731df6174 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 2, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:16.948109 30500 catalog_manager.cc:5649] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 3, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:16.974475 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.978423 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.979080 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.938462 31855 raft_consensus.cc:2804] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:16.980518 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:16.981053 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:16.937484 31932 raft_consensus.cc:2749] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:16.930750 32022 raft_consensus.cc:2749] T 4614bdb7f750431fa840cabccf782b4b P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:16.994314 32025 raft_consensus.cc:2272] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:04:16.923099 31812 consensus_queue.cc:1059] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20251025 20:04:16.946417 30738 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f1269e0f1364a69b9a36ce877b0da5e with delete_type TABLET_DATA_DELETED (Replaced by 6164729145ad47d6bbc2e7b3c6b49a22 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:53746
I20251025 20:04:16.988694 30745 raft_consensus.cc:2468] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 6.
I20251025 20:04:17.021461 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:16.990505 30677 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 6
I20251025 20:04:16.989382 30671 raft_consensus.cc:1240] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:16.943593 31801 raft_consensus.cc:2749] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:16.993474 30678 raft_consensus.cc:1275] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251025 20:04:16.999285 30567 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:16.999361 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:16.919368 31885 consensus_queue.cc:1048] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20251025 20:04:16.982349 30645 leader_election.cc:400] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 6.
I20251025 20:04:17.003268 30590 tablet_service.cc:1505] Processing CreateTablet for tablet 01d56eb384124dfcb1bcd35d77e89f0e (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251025 20:04:16.972700 30500 catalog_manager.cc:5649] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 3 to 4, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 4 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:17.009706 30592 tablet_service.cc:1505] Processing CreateTablet for tablet 6164729145ad47d6bbc2e7b3c6b49a22 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251025 20:04:17.012552 30739 tablet_service.cc:1505] Processing CreateTablet for tablet 7294f698d29e4c919745b460e4e25eeb (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251025 20:04:17.009168 30589 tablet_service.cc:1505] Processing CreateTablet for tablet d8afaad4899545a9a4169026b63c0699 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251025 20:04:17.014981 30742 tablet_service.cc:1505] Processing CreateTablet for tablet 01d56eb384124dfcb1bcd35d77e89f0e (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251025 20:04:16.947129 31754 raft_consensus.cc:2749] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:16.998654 32030 raft_consensus.cc:3060] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 6
I20251025 20:04:16.930121 31876 consensus_queue.cc:1048] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20251025 20:04:16.947409 31942 raft_consensus.cc:2804] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:17.021656 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:16.924788 32021 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:16.998477 31855 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:16.946902 31895 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:17.028301 30567 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.023311 31935 raft_consensus.cc:2749] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:17.014925 30736 tablet_service.cc:1505] Processing CreateTablet for tablet f69f57fe417f492d9b82ed651b9d70c1 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251025 20:04:17.028918 30500 catalog_manager.cc:5649] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 3 to 4, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 4 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:16.993719 30734 tablet_service.cc:1552] Processing DeleteTablet for tablet 580e75e8dcd949e39aff92ada50b40be with delete_type TABLET_DATA_DELETED (Replaced by 2d6dd59f914c41758a18803991215caf at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:53746
I20251025 20:04:17.085232 30739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7294f698d29e4c919745b460e4e25eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:17.026073 30570 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:17.053321 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 58463.7us
I20251025 20:04:16.947175 31871 raft_consensus.cc:2749] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:17.059046 30750 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 6.
I20251025 20:04:17.056563 31942 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:17.017012 30735 tablet_service.cc:1505] Processing CreateTablet for tablet d8afaad4899545a9a4169026b63c0699 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251025 20:04:17.075554 31895 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
W20251025 20:04:17.064757 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 52115.6us
I20251025 20:04:17.074139 30677 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 6.
I20251025 20:04:17.027233 31679 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.081154 30500 catalog_manager.cc:5649] T 06d7fa7a698c4890af235aed97fd6fb5 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:17.079483 31850 raft_consensus.cc:2749] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251025 20:04:17.052799 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 70760.3us
I20251025 20:04:17.092947 32025 ts_tablet_manager.cc:1916] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:04:17.105921 31769 raft_consensus.cc:2749] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251025 20:04:17.098363 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20251025 20:04:17.077805 32036 log.cc:927] Time spent T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20251025 20:04:16.998301 31797 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:17.131794 32021 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.131664 31797 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.132802 30742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01d56eb384124dfcb1bcd35d77e89f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:17.134294 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:17.135119 30752 raft_consensus.cc:2393] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:17.136030 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:17.137869 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:17.106508 31942 raft_consensus.cc:3060] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 2
W20251025 20:04:17.103272 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 75066.9us
I20251025 20:04:17.063978 31855 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:17.142081 30567 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.023727 31939 consensus_queue.cc:1048] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.072s
I20251025 20:04:17.024288 31567 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:17.106086 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 80235.1us
I20251025 20:04:17.073206 32033 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:17.100703 30590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01d56eb384124dfcb1bcd35d77e89f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:16.951296 32034 consensus_queue.cc:1059] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.142s
I20251025 20:04:17.117483 30740 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d9831caf5d44fc080590d41bd68a083 with delete_type TABLET_DATA_DELETED (Replaced by 5ee1ba6e04b74e1faf5c9819144cd740 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:53746
I20251025 20:04:17.111295 30500 catalog_manager.cc:5649] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:17.216449 30591 tablet_service.cc:1505] Processing CreateTablet for tablet 5ee1ba6e04b74e1faf5c9819144cd740 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251025 20:04:17.129053 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:17.098971 32011 consensus_queue.cc:237] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.217873 30500 catalog_manager.cc:5649] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 4 to 5, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 5 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:17.219909 30500 catalog_manager.cc:5649] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 2 to 3, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:17.023468 31794 consensus_queue.cc:1048] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
W20251025 20:04:17.129066 30645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:17.079728 31485 consensus_queue.cc:1048] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251025 20:04:17.112491 31801 raft_consensus.cc:2804] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election won for term 6
I20251025 20:04:17.147229 31997 raft_consensus.cc:2749] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:17.152455 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:04:17.202363 32009 log.cc:927] Time spent T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20251025 20:04:17.157171 32021 consensus_queue.cc:1048] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20251025 20:04:17.144392 30567 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:17.146476 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:17.141263 32023 raft_consensus.cc:2243] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:04:17.206712 30570 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:17.133816 31797 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:17.213310 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 98970.2us
I20251025 20:04:17.214054 31771 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:17.211382 30588 tablet_service.cc:1505] Processing CreateTablet for tablet 8360e93e9b6d471791c2bd59bf8292ab (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251025 20:04:17.201961 32030 raft_consensus.cc:2749] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 6.
I20251025 20:04:17.216148 31742 consensus_queue.cc:1048] T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:04:17.223620 30645 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.224021 30500 catalog_manager.cc:5649] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 2, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:17.231431 30746 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:17.266609 30645 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.266673 30500 catalog_manager.cc:5649] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 2, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:17.140231 31940 consensus_queue.cc:1059] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
W20251025 20:04:17.237761 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 93466.1us
W20251025 20:04:17.224771 32038 log.cc:927] Time spent T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20251025 20:04:17.155197 31947 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:17.233801 32028 log.cc:927] Time spent T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20251025 20:04:17.237396 31801 raft_consensus.cc:697] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [term 6 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
W20251025 20:04:17.237823 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 90727.9us
I20251025 20:04:17.242022 30670 raft_consensus.cc:3055] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:17.148530 31567 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.225875 32033 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.268939 30500 catalog_manager.cc:5649] T d4af0795c8ff48d9b639058ea7111f12 P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 2, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:17.276469 30567 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:17.270382 31885 raft_consensus.cc:2804] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election won for term 6
I20251025 20:04:17.301530 30500 catalog_manager.cc:5649] T bc001e0fba604d68bfa98fc25d70bbbb P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:17.265533 31771 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.289144 30679 raft_consensus.cc:2468] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 6.
I20251025 20:04:17.264479 31850 raft_consensus.cc:2749] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:17.305686 30719 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:17.306299 30600 raft_consensus.cc:1240] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:17.271195 32029 raft_consensus.cc:2749] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:17.282227 30670 raft_consensus.cc:740] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:17.282181 30746 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:17.307324 30500 catalog_manager.cc:5649] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e reported cstate change: term changed from 1 to 2, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 2 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:17.313680 30670 consensus_queue.cc:260] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.315404 30670 raft_consensus.cc:3060] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:17.320286 30570 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:17.319252 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:17.325271 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 48897.5us
I20251025 20:04:17.327064 31962 consensus_queue.cc:1059] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:04:17.328095 31812 raft_consensus.cc:2804] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election won for term 6
I20251025 20:04:17.328641 31812 raft_consensus.cc:697] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [term 6 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:17.329236 31962 raft_consensus.cc:3055] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:17.329665 30592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6164729145ad47d6bbc2e7b3c6b49a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:17.330137 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:17.331354 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:17.335899 30672 raft_consensus.cc:3060] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:17.332010 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:17.340426 30670 raft_consensus.cc:1275] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20251025 20:04:17.331846 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:17.340867 30670 pending_rounds.cc:85] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 1
I20251025 20:04:17.341120 30602 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 5.
I20251025 20:04:17.341264 30670 pending_rounds.cc:107] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251025 20:04:17.341542 30670 raft_consensus.cc:2889] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:04:17.294116 32011 raft_consensus.cc:2749] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:17.342929 30598 raft_consensus.cc:2368] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:17.347440 31801 consensus_queue.cc:237] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.348160 31567 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.329663 31962 raft_consensus.cc:740] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:17.350539 32034 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20251025 20:04:17.350176 31962 consensus_queue.cc:260] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.351408 31962 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:17.352039 32033 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.357285 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:17.364929 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:17.365617 30674 raft_consensus.cc:2468] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:17.319658 31998 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:17.311269 31946 consensus_queue.cc:1048] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20251025 20:04:17.335798 30749 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
W20251025 20:04:17.378237 31847 consensus_meta.cc:287] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.656s user 0.004s sys 0.007s
W20251025 20:04:17.402386 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 135849us
I20251025 20:04:17.403739 30645 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.404752 30645 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.405627 30645 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.406744 30645 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:17.376528 31751 raft_consensus.cc:2804] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:17.377651 31797 raft_consensus.cc:2804] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:17.329536 31812 consensus_queue.cc:237] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.411978 30645 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.364121 31885 raft_consensus.cc:697] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [term 6 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:17.423637 30645 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.418020 30589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8afaad4899545a9a4169026b63c0699. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:17.414988 32033 raft_consensus.cc:2749] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:17.415331 32011 raft_consensus.cc:2749] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:17.418464 31941 raft_consensus.cc:2804] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:17.399520 31414 consensus_queue.cc:1048] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20251025 20:04:17.459508 30672 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:17.393981 31998 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.435539 31797 raft_consensus.cc:697] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:17.434690 31945 consensus_queue.cc:1048] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20251025 20:04:17.453116 31941 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:17.476083 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 61606.4us
I20251025 20:04:17.475896 30716 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:17.458777 31876 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:17.470250 30588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8360e93e9b6d471791c2bd59bf8292ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:17.484189 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:17.490648 30594 tablet_service.cc:1505] Processing CreateTablet for tablet 2d6dd59f914c41758a18803991215caf (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251025 20:04:17.492245 31771 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.470428 32011 raft_consensus.cc:2749] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:17.494572 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:17.502705 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:17.503750 30746 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:17.505456 30570 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:17.542904 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 121276us
I20251025 20:04:17.543926 30645 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.481451 31941 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:17.489291 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:17.474104 32052 raft_consensus.cc:2749] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251025 20:04:17.477253 31947 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.495462 30747 raft_consensus.cc:3060] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:17.552146 31849 raft_consensus.cc:2804] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:17.564812 31849 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:17.568575 30594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d6dd59f914c41758a18803991215caf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 20:04:17.553130 32012 log.cc:927] Time spent T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20251025 20:04:17.470779 32035 consensus_queue.cc:1059] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
I20251025 20:04:17.575598 30645 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:17.584008 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:17.588340 32051 tablet_bootstrap.cc:492] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
W20251025 20:04:17.594321 32049 log.cc:927] Time spent T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20251025 20:04:17.569802 31849 raft_consensus.cc:3060] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:17.486280 31995 consensus_queue.cc:1048] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20251025 20:04:17.487772 31876 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.570003 31797 consensus_queue.cc:237] T da4360132dff42a18667801272f14797 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.610144 32053 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:17.586758 31895 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.611115 32056 raft_consensus.cc:2804] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:17.566255 31771 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:17.616906 30645 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
W20251025 20:04:17.573437 32004 log.cc:927] Time spent T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251025 20:04:17.481637 31885 consensus_queue.cc:237] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.634546 31993 consensus_queue.cc:1048] T b0a9871641274e83b2300f3ad8095add P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:17.607859 31940 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:17.644172 32052 raft_consensus.cc:2749] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:17.561318 32013 consensus_queue.cc:1048] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.087s
I20251025 20:04:17.655260 32056 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:17.615289 31942 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.669345 30591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ee1ba6e04b74e1faf5c9819144cd740. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:17.691746 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:17.664016 31751 raft_consensus.cc:697] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
W20251025 20:04:17.710675 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:17.668157 32053 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.725843 30671 raft_consensus.cc:1240] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:17.689754 32011 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:17.681432 31998 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:17.691948 32022 raft_consensus.cc:932] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 1 LEADER]: Unable to handle notification of new term (2): Illegal state: RaftConsensus is not running: State = Stopping
I20251025 20:04:17.728865 31751 consensus_queue.cc:237] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:17.737217 32045 log.cc:927] Time spent T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.268s user 0.002s sys 0.000s
I20251025 20:04:17.660463 31771 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.689321 32056 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:17.670024 31940 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.759379 30598 raft_consensus.cc:1240] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:17.692765 31945 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:17.737711 32050 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:17.751662 32051 tablet_bootstrap.cc:654] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:17.788367 30602 raft_consensus.cc:2468] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:17.734438 32011 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.773820 32062 raft_consensus.cc:2804] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:17.840799 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:17.841885 30676 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:17.843091 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:17.847278 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:17.859493 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:17.860615 30676 raft_consensus.cc:2468] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:17.785535 31945 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:17.843796 31948 consensus_meta.cc:287] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 1.043s user 0.006s sys 0.011s
I20251025 20:04:17.868180 30716 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:17.775362 30741 tablet_service.cc:1505] Processing CreateTablet for tablet 2d6dd59f914c41758a18803991215caf (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251025 20:04:17.846130 32062 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:17.853519 31567 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:17.856211 31876 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:17.859004 31895 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.878695 31992 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:17.891633 31962 consensus_meta.cc:287] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.540s user 0.013s sys 0.000s
I20251025 20:04:17.893513 31414 raft_consensus.cc:2804] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251025 20:04:17.904749 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 119998us
I20251025 20:04:17.912035 32071 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:17.881242 31948 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.915628 32071 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.774639 32050 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.917651 30601 raft_consensus.cc:1240] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
W20251025 20:04:17.919782 32063 log.cc:927] Time spent T 43d8dfe478a4430da93c214b20963f9f P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20251025 20:04:17.920652 32071 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.889766 31744 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:17.890785 32062 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:17.915515 31771 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.943851 31945 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:17.917865 31414 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:17.924103 30740 tablet_service.cc:1505] Processing CreateTablet for tablet 6164729145ad47d6bbc2e7b3c6b49a22 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251025 20:04:17.944553 31809 consensus_queue.cc:1059] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 20:04:17.935330 31744 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.952481 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:17.946332 32057 log.cc:927] Time spent T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.214s user 0.002s sys 0.000s
I20251025 20:04:17.954787 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:17.955413 30596 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:17.898768 31567 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.962055 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:17.925415 31942 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.930013 32053 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.955925 31414 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:17.989642 30719 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:17.975246 31744 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:17.993196 30598 raft_consensus.cc:2468] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:17.999660 32013 raft_consensus.cc:2804] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:17.957588 30738 tablet_service.cc:1505] Processing CreateTablet for tablet 5ee1ba6e04b74e1faf5c9819144cd740 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251025 20:04:18.003979 30747 raft_consensus.cc:2468] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 6.
I20251025 20:04:17.985319 31948 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:17.918442 31992 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:17.986913 31940 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.004827 32013 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:17.946655 30734 tablet_service.cc:1505] Processing CreateTablet for tablet 8360e93e9b6d471791c2bd59bf8292ab (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
W20251025 20:04:18.032445 31855 consensus_meta.cc:287] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.833s user 0.011s sys 0.002s
I20251025 20:04:18.032959 31855 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.034533 31992 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:18.040717 30673 raft_consensus.cc:1240] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:18.017977 32011 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.043429 32050 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.047413 31876 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:18.050977 31876 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.058583 30679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41100 (request call id 505) took 1081 ms. Trace:
I20251025 20:04:18.059017 30679 rpcz_store.cc:276] 1025 20:04:16.977646 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.979340 (+ 1694us) service_pool.cc:225] Handling call
1025 20:04:18.058414 (+1079074us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b540007bf54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:18.058523 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":662031232}
I20251025 20:04:18.048803 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:18.062749 30596 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:18.073354 30567 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:18.040006 32013 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:18.078747 30719 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:18.084380 31992 raft_consensus.cc:2804] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:18.084857 31992 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:18.087145 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:18.087971 30596 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:18.089347 30719 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:18.081929 31855 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.080745 32030 raft_consensus.cc:3055] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:18.086251 31992 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:18.085951 31876 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:18.131160 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 81272.5us
I20251025 20:04:18.132368 30567 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:18.113201 30736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69f57fe417f492d9b82ed651b9d70c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:18.105084 32044 raft_consensus.cc:2749] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:18.097930 31485 raft_consensus.cc:2804] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:18.097764 32029 raft_consensus.cc:2749] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:18.112628 31855 raft_consensus.cc:493] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:18.141685 31941 consensus_meta.cc:287] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.573s user 0.009s sys 0.003s
@ 0x7fd81ec5dd49 testing::TestInfo::Run() at ??:0
I20251025 20:04:18.109396 32030 raft_consensus.cc:740] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:18.205157 31855 raft_consensus.cc:515] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.158324 31414 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.207365 31892 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:18.208400 30489 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 48618.9us
W20251025 20:04:18.212409 31849 consensus_meta.cc:287] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.608s user 0.016s sys 0.003s
I20251025 20:04:18.261257 31948 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:18.261765 31849 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.214833 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:18.217092 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:18.217500 30604 raft_consensus.cc:1240] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251025 20:04:18.217829 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:18.263644 30678 raft_consensus.cc:2468] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:18.218739 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:18.219444 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:18.136353 31876 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:18.254698 32044 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:18.260742 31892 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.269941 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:18.265197 30679 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:18.270738 30674 raft_consensus.cc:2468] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:18.266589 30567 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:18.261955 31948 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.277945 30567 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:18.279810 30567 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:18.157513 31485 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:18.282466 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:18.278647 31849 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:18.273290 31876 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.286650 31876 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:18.286700 31414 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:18.289112 31892 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:18.290896 32022 raft_consensus.cc:2804] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:18.291322 32022 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:18.291584 31626 raft_consensus.cc:3055] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:18.220495 31567 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.294493 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:18.295123 30604 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:18.300079 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:18.308744 30645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.016s sys 0.028s
W20251025 20:04:18.309441 30645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s user 0.016s sys 0.028s
I20251025 20:04:18.309937 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:18.310091 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 693245us
I20251025 20:04:18.310909 30596 raft_consensus.cc:2468] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:18.311170 30645 leader_election.cc:304] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:18.292100 31679 raft_consensus.cc:2804] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:18.291683 32022 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:18.296873 31485 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
W20251025 20:04:18.359498 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 46679.1us
I20251025 20:04:18.297552 31849 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:18.292140 31626 raft_consensus.cc:740] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:18.362617 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 52612.1us
W20251025 20:04:18.314651 31812 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.012s sys 0.027s
I20251025 20:04:18.365104 30645 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:18.260730 31941 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.367672 30719 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:18.292289 31754 raft_consensus.cc:2804] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:18.374594 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:18.376555 30719 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:18.361122 30741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d6dd59f914c41758a18803991215caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:18.375540 30601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 800) took 1573 ms. Trace:
I20251025 20:04:18.351518 32029 raft_consensus.cc:2749] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:18.377717 30601 rpcz_store.cc:276] 1025 20:04:16.802526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.806085 (+ 3559us) service_pool.cc:225] Handling call
1025 20:04:16.806483 (+ 398us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:17.923500 (+1117017us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:18.375217 (+451717us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400040954. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:18.375470 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2554966784}
I20251025 20:04:18.297675 31993 consensus_queue.cc:1059] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20251025 20:04:18.360638 31679 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:18.407250 31876 consensus_queue.cc:1059] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:18.279093 32044 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:18.377400 31812 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s user 0.013s sys 0.028s
I20251025 20:04:18.380338 31754 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:18.383247 32035 raft_consensus.cc:2749] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:18.379367 32050 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:18.414979 31679 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:18.404014 31414 raft_consensus.cc:2804] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251025 20:04:18.431003 30742 consensus_meta.cc:287] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.710s user 0.008s sys 0.015s
I20251025 20:04:18.429848 32070 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:18.445413 32056 consensus_meta.cc:287] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.665s user 0.006s sys 0.013s
I20251025 20:04:18.431313 31934 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:18.374513 31849 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.408501 31970 raft_consensus.cc:2804] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:18.418504 31948 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.348496 32030 consensus_queue.cc:260] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.459120 31414 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:18.449649 31754 raft_consensus.cc:3060] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:18.476788 32056 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.482897 31941 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.475083 32070 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:18.486898 32062 consensus_meta.cc:287] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.531s user 0.009s sys 0.003s
I20251025 20:04:18.487339 31855 leader_election.cc:290] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.480921 31934 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.489235 30742 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 69) took 1477 ms. Trace:
I20251025 20:04:18.511255 31849 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:18.512996 32056 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.518725 31485 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.521723 31934 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:18.542709 31626 consensus_queue.cc:260] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.544292 31626 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:18.501633 31414 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:18.549448 32030 raft_consensus.cc:3060] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 6
I20251025 20:04:18.553256 32013 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.555600 32013 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:18.557814 31679 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.563110 32023 raft_consensus.cc:2272] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:04:18.564085 31679 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.506702 30742 rpcz_store.cc:276] 1025 20:04:17.011559 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.013842 (+ 2283us) service_pool.cc:225] Handling call
1025 20:04:17.075194 (+ 61352us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:17.100356 (+ 25162us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:17.645135 (+544779us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:18.476485 (+831350us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:18.489148 (+ 12663us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:18.460999 32050 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.566172 32050 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.508414 32062 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:18.547238 30749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 519) took 2498 ms (client timeout 1508 ms). Trace:
W20251025 20:04:18.567389 30749 rpcz_store.cc:269] 1025 20:04:16.049362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.054144 (+ 4782us) service_pool.cc:225] Handling call
1025 20:04:16.530712 (+476568us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea507ce 000000000035a3c9 00007fd826a6d030 00007fd826b35864 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:18.547009 (+2016297us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b5400209c54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:18.547136 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":791332480}
I20251025 20:04:18.568619 32062 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.527771 32044 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.539691 31485 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:18.526376 32070 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.576289 32025 ts_tablet_manager.cc:1929] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 20:04:18.476881 31970 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:18.600131 31970 raft_consensus.cc:3060] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:18.594488 31992 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.606467 31992 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:18.597827 32023 ts_tablet_manager.cc:1916] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:04:18.611430 32025 log.cc:1199] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: Deleting WAL directory at /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-2-root/wals/e8c971a6d4854180b4d243fa79fbbc11
I20251025 20:04:18.613839 32025 ts_tablet_manager.cc:1950] T e8c971a6d4854180b4d243fa79fbbc11 P 783f8fd6d80141b7912f66a954a6f92e: Deleting consensus metadata
I20251025 20:04:18.614966 32022 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.629297 32025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53746 (request call id 59) took 2108 ms. Trace:
I20251025 20:04:18.631057 32025 rpcz_store.cc:276] 1025 20:04:16.520823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.522030 (+ 1207us) service_pool.cc:225] Handling call
1025 20:04:16.699854 (+177824us) ts_tablet_manager.cc:1151] Deleting tablet e8c971a6d4854180b4d243fa79fbbc11
1025 20:04:16.699987 (+ 133us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 20:04:18.050782 (+1350795us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:18.575905 (+525123us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:18.629168 (+ 53263us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":229812,"tablet-delete.queue_time_us":174647,"thread_start_us":108310,"threads_started":1}
I20251025 20:04:18.633493 32025 tablet_replica.cc:333] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: stopping tablet replica
W20251025 20:04:18.680732 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:18.681839 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:18.687588 30604 raft_consensus.cc:2393] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:18.691336 30600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 803) took 1879 ms. Trace:
I20251025 20:04:18.691977 30600 rpcz_store.cc:276] 1025 20:04:16.811415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.812813 (+ 1398us) service_pool.cc:225] Handling call
1025 20:04:16.813200 (+ 387us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:17.306781 (+493581us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:18.690489 (+1383708us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b5400311ad4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:18.691043 (+ 554us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1133536000}
I20251025 20:04:18.698585 30662 tablet_service.cc:1552] Processing DeleteTablet for tablet e8c971a6d4854180b4d243fa79fbbc11 with delete_type TABLET_DATA_DELETED (Replaced by f69f57fe417f492d9b82ed651b9d70c1 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:41086
I20251025 20:04:18.718328 30734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8360e93e9b6d471791c2bd59bf8292ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:18.763471 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:18.766453 30679 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 5
I20251025 20:04:18.784749 32022 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.785840 32044 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:18.805027 32030 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:18.805960 32030 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.809284 32030 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.810082 32070 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:18.810308 32011 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:18.810989 32070 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.815681 30678 raft_consensus.cc:1240] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:18.826798 30735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8afaad4899545a9a4169026b63c0699. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:18.828366 30671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 812) took 1835 ms. Trace:
I20251025 20:04:18.829114 30671 rpcz_store.cc:276] 1025 20:04:16.993137 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.024390 (+ 31253us) service_pool.cc:225] Handling call
1025 20:04:17.024684 (+ 294us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:17.726539 (+701855us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:18.827687 (+1101148us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54003b1fd4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:18.828121 (+ 434us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1611843328}
I20251025 20:04:18.835961 32044 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.894156 32070 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:18.811096 32011 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:18.968395 30642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.271s user 0.022s sys 0.061s
W20251025 20:04:18.969003 30642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.294s user 0.023s sys 0.061s
W20251025 20:04:18.970103 32051 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 1.106s user 0.023s sys 0.070s
W20251025 20:04:18.970636 32051 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 1.113s user 0.024s sys 0.070s
W20251025 20:04:18.971165 30597 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.015s user 0.026s sys 0.060s
W20251025 20:04:18.971524 30597 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.016s user 0.026s sys 0.060s
I20251025 20:04:18.972842 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:18.973735 30671 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 4
I20251025 20:04:18.974009 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:18.974887 30674 raft_consensus.cc:2468] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:18.976413 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:18.977780 30670 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.271s user 0.006s sys 0.043s
W20251025 20:04:18.978201 30670 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.315s user 0.007s sys 0.043s
I20251025 20:04:18.978240 30664 tablet_service.cc:1552] Processing DeleteTablet for tablet fea0c74c9c0c4a69afe072c45c4cbe99 with delete_type TABLET_DATA_DELETED (Replaced by 7294f698d29e4c919745b460e4e25eeb at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:41086
I20251025 20:04:18.979656 30740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6164729145ad47d6bbc2e7b3c6b49a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:18.981657 32062 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:18.982538 32062 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:18.983259 30673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 810) took 1990 ms. Trace:
I20251025 20:04:18.983615 30673 rpcz_store.cc:276] 1025 20:04:16.992663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.009485 (+ 16822us) service_pool.cc:225] Handling call
1025 20:04:17.009783 (+ 298us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:18.042990 (+1033207us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:18.982674 (+939684us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54001cf7d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:18.983066 (+ 392us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2368268672}
W20251025 20:04:18.984760 30570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.251s user 0.028s sys 0.052s
W20251025 20:04:18.985146 30570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.297s user 0.028s sys 0.052s
I20251025 20:04:18.985038 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:19.008458 30741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 74) took 1263 ms. Trace:
I20251025 20:04:19.010178 30741 rpcz_store.cc:276] 1025 20:04:17.745008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.768815 (+ 23807us) service_pool.cc:225] Handling call
1025 20:04:18.131979 (+363164us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:18.157784 (+ 25805us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:18.803911 (+646127us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:19.007140 (+203229us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:19.008161 (+ 1021us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:19.013417 30567 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:19.031749 31679 raft_consensus.cc:2804] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:19.032358 31679 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.032822 31679 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:19.036444 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:19.014139 31849 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:19.039963 31849 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.042694 31849 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:19.015496 30678 raft_consensus.cc:1240] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:19.018555 30588 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 72) took 1809 ms. Trace:
I20251025 20:04:19.019790 31993 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:19.045079 31993 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.044824 30588 rpcz_store.cc:276] 1025 20:04:17.209310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.210104 (+ 794us) service_pool.cc:225] Handling call
1025 20:04:17.345753 (+135649us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:17.345913 (+ 160us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:18.677626 (+1331713us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:19.013317 (+335691us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:19.018211 (+ 4894us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:19.020241 30602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34874 (request call id 806) took 1437 ms (client timeout 1801 ms). Trace:
I20251025 20:04:19.020594 31485 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:19.048797 31414 consensus_meta.cc:287] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.502s user 0.008s sys 0.004s
I20251025 20:04:19.049302 31485 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.052138 31485 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:19.024914 31797 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.040s user 0.038s sys 0.087s
W20251025 20:04:19.052995 31797 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.143s user 0.039s sys 0.088s
I20251025 20:04:19.053172 30673 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 3
W20251025 20:04:19.048592 30602 rpcz_store.cc:269] 1025 20:04:17.582333 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.583545 (+ 1212us) service_pool.cc:225] Handling call
1025 20:04:19.019711 (+1436166us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b5400041ad4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:19.019950 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":459122560}
I20251025 20:04:19.048162 31993 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:19.055816 32019 raft_consensus.cc:3055] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:19.056452 32019 raft_consensus.cc:740] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:19.058758 32019 consensus_queue.cc:260] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.062424 31941 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:19.062238 32021 consensus_queue.cc:1059] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:19.063174 31941 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.064260 32070 raft_consensus.cc:3055] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:19.064718 32070 raft_consensus.cc:740] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:19.062296 32072 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:18.949254 31801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.437s user 0.045s sys 0.080s
W20251025 20:04:18.967967 30595 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.147s user 0.012s sys 0.021s
I20251025 20:04:19.065610 32072 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.079083 30664 tablet_service.cc:1552] Processing DeleteTablet for tablet da8a0eaa68044835a4cf94340ca25ac5 with delete_type TABLET_DATA_DELETED (Replaced by 01d56eb384124dfcb1bcd35d77e89f0e at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:41086
I20251025 20:04:19.049405 31414 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.081187 32072 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.082289 31414 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:19.085346 31941 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.066155 32070 consensus_queue.cc:260] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.071596 30679 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:19.090304 32070 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
W20251025 20:04:19.115072 30595 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.294s user 0.013s sys 0.021s
W20251025 20:04:19.115665 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:19.113531 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:19.069186 32019 raft_consensus.cc:3060] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:19.121263 30671 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:19.112758 30592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 71) took 2104 ms. Trace:
I20251025 20:04:19.115422 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:19.110028 31801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.614s user 0.045s sys 0.081s
W20251025 20:04:19.130127 32015 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.164s user 0.005s sys 0.038s
W20251025 20:04:19.130679 32015 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.366s user 0.005s sys 0.038s
I20251025 20:04:19.131206 30567 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:19.133920 30678 pending_rounds.cc:85] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 1
I20251025 20:04:19.112848 32044 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.144719 31941 raft_consensus.cc:493] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:19.142899 30678 pending_rounds.cc:107] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20251025 20:04:19.151860 30678 raft_consensus.cc:2889] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:04:19.153239 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:19.181648 31414 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:19.182651 31414 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.185495 31414 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:19.127198 30592 rpcz_store.cc:276] 1025 20:04:17.008255 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.008510 (+ 255us) service_pool.cc:225] Handling call
1025 20:04:17.026797 (+ 18287us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:17.328925 (+302128us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b5800000f98. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50bee 00007fd827912e89 00007fd826a6cca1 00007fd826b2f469 00007fd826addf59 00007fd81ff83bf4 00007fd81ff83b82 00007fd81ff83ab1 00007fd81ff83a2c
1025 20:04:17.329059 (+ 134us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:18.681668 (+1352609us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:19.022086 (+340418us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:19.112489 (+ 90403us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3734144}
I20251025 20:04:19.186654 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:19.188733 31754 consensus_meta.cc:287] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.658s user 0.005s sys 0.009s
I20251025 20:04:19.188905 32051 tablet_bootstrap.cc:492] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:04:19.189270 32050 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:19.163164 30664 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e7a1b97c4aa4812a26b178e32476392 with delete_type TABLET_DATA_DELETED (Replaced by d8afaad4899545a9a4169026b63c0699 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:41086
I20251025 20:04:19.161587 32062 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:19.189832 30739 thread.cc:641] tablet-open [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.014s sys 0.008s
W20251025 20:04:19.218076 30739 thread.cc:608] tablet-open [worker] (thread pool) Time spent starting thread: real 0.844s user 0.014s sys 0.009s
I20251025 20:04:19.219137 30739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 68) took 2208 ms. Trace:
I20251025 20:04:19.219676 30739 rpcz_store.cc:276] 1025 20:04:17.011145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.011380 (+ 235us) service_pool.cc:225] Handling call
1025 20:04:17.033108 (+ 21728us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:17.056537 (+ 23429us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:17.996867 (+940330us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:18.141786 (+144919us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:19.218990 (+1077204us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":847867,"threads_started":1}
I20251025 20:04:19.246543 30668 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f1269e0f1364a69b9a36ce877b0da5e with delete_type TABLET_DATA_DELETED (Replaced by 6164729145ad47d6bbc2e7b3c6b49a22 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:41086
W20251025 20:04:19.247606 31871 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.022s sys 0.036s
W20251025 20:04:19.248139 31871 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.963s user 0.023s sys 0.037s
I20251025 20:04:19.251601 30673 raft_consensus.cc:2468] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
W20251025 20:04:19.226500 30677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.037s user 0.011s sys 0.016s
W20251025 20:04:19.206292 31751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.147s user 0.025s sys 0.082s
W20251025 20:04:19.226135 30596 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.015s sys 0.004s
W20251025 20:04:19.302168 30677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.224s user 0.011s sys 0.016s
W20251025 20:04:19.399950 30596 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.981s user 0.016s sys 0.005s
W20251025 20:04:19.414324 30598 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.048s user 0.018s sys 0.036s
W20251025 20:04:19.209945 31885 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.120s user 0.005s sys 0.041s
W20251025 20:04:19.416877 32075 log.cc:927] Time spent T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.195s user 0.000s sys 0.000s
I20251025 20:04:19.254758 31801 raft_consensus.cc:2804] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:19.260251 32062 raft_consensus.cc:493] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:19.286836 31751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.340s user 0.027s sys 0.082s
I20251025 20:04:19.429253 32073 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.430721 32072 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.232235 32022 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.194576 31941 raft_consensus.cc:515] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:19.255623 30716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.258s user 0.023s sys 0.044s
I20251025 20:04:19.253927 31871 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:19.464433 30598 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.099s user 0.018s sys 0.039s
W20251025 20:04:19.405583 30717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.736s user 0.014s sys 0.009s
W20251025 20:04:19.447011 30603 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.461s user 0.022s sys 0.057s
I20251025 20:04:19.468218 30738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ee1ba6e04b74e1faf5c9819144cd740. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 20:04:19.471792 30718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:19.475137 30664 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d9831caf5d44fc080590d41bd68a083 with delete_type TABLET_DATA_DELETED (Replaced by 5ee1ba6e04b74e1faf5c9819144cd740 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:41086
I20251025 20:04:19.477819 30665 tablet_service.cc:1552] Processing DeleteTablet for tablet 580e75e8dcd949e39aff92ada50b40be with delete_type TABLET_DATA_DELETED (Replaced by 2d6dd59f914c41758a18803991215caf at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:41086
I20251025 20:04:19.477928 30664 tablet_service.cc:1505] Processing CreateTablet for tablet 7294f698d29e4c919745b460e4e25eeb (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251025 20:04:19.480083 30661 tablet_service.cc:1505] Processing CreateTablet for tablet f69f57fe417f492d9b82ed651b9d70c1 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251025 20:04:19.480561 30664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7294f698d29e4c919745b460e4e25eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:19.481604 30661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69f57fe417f492d9b82ed651b9d70c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:19.483301 30667 tablet_service.cc:1505] Processing CreateTablet for tablet 01d56eb384124dfcb1bcd35d77e89f0e (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251025 20:04:19.484454 31941 leader_election.cc:290] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.485087 31871 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.485791 30669 tablet_service.cc:1505] Processing CreateTablet for tablet d8afaad4899545a9a4169026b63c0699 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251025 20:04:19.487403 31871 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.207656 31754 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.462824 30734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 72) took 1789 ms. Trace:
W20251025 20:04:19.507933 30716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.621s user 0.027s sys 0.044s
W20251025 20:04:19.492208 30662 thread.cc:641] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.792s user 0.027s sys 0.066s
W20251025 20:04:19.510387 30662 thread.cc:608] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.810s user 0.029s sys 0.067s
I20251025 20:04:19.510565 31941 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.511229 31941 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.511677 31754 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.512905 32062 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.513602 31941 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:19.493062 30717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.084s user 0.015s sys 0.009s
W20251025 20:04:19.514645 30603 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.840s user 0.023s sys 0.057s
I20251025 20:04:19.516237 30668 tablet_service.cc:1552] Processing DeleteTablet for tablet 4614bdb7f750431fa840cabccf782b4b with delete_type TABLET_DATA_DELETED (Replaced by 8360e93e9b6d471791c2bd59bf8292ab at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:41086
W20251025 20:04:19.518797 30589 consensus_meta.cc:287] T d8afaad4899545a9a4169026b63c0699 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 1.310s user 0.013s sys 0.004s
I20251025 20:04:19.460247 32050 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:19.528658 30589 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34862 (request call id 70) took 2552 ms. Trace:
I20251025 20:04:19.530112 32050 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:19.532013 32098 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:19.532662 32098 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.532871 30660 tablet_service.cc:1505] Processing CreateTablet for tablet 6164729145ad47d6bbc2e7b3c6b49a22 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251025 20:04:19.537395 31414 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:19.537539 31993 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:19.465783 32011 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.529176 30589 rpcz_store.cc:276] 1025 20:04:16.976250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.007825 (+ 31575us) service_pool.cc:225] Handling call
1025 20:04:17.332009 (+324184us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:17.342529 (+ 10520us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:18.208313 (+865784us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:19.527834 (+1319521us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:19.528583 (+ 749us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:19.524358 31801 raft_consensus.cc:697] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [term 5 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:19.549157 32051 ts_tablet_manager.cc:1403] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 2.037s user 0.036s sys 0.085s
I20251025 20:04:19.588253 30604 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
W20251025 20:04:19.463536 31885 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.504s user 0.010s sys 0.042s
I20251025 20:04:19.504943 30734 rpcz_store.cc:276] 1025 20:04:17.673811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.682003 (+ 8192us) service_pool.cc:225] Handling call
1025 20:04:18.269103 (+587100us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:18.284291 (+ 15188us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:19.135374 (+851083us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:19.251689 (+116315us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:19.462731 (+211042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:19.611438 30735 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 70) took 2599 ms. Trace:
I20251025 20:04:19.611769 30489 catalog_manager.cc:4985] TS 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): tablet e8c971a6d4854180b4d243fa79fbbc11 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) successfully deleted
I20251025 20:04:19.612022 30735 rpcz_store.cc:276] 1025 20:04:17.011904 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.015838 (+ 3934us) service_pool.cc:225] Handling call
1025 20:04:18.112428 (+1096590us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:18.131366 (+ 18938us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:19.142363 (+1010997us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:19.536837 (+394474us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:19.611323 (+ 74486us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:19.614049 32023 ts_tablet_manager.cc:1929] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 20:04:19.614857 32023 log.cc:1199] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: Deleting WAL directory at /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-0-root/wals/e8c971a6d4854180b4d243fa79fbbc11
I20251025 20:04:19.538107 31414 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.596190 31993 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:19.630221 30668 tablet_service.cc:1505] Processing CreateTablet for tablet 8360e93e9b6d471791c2bd59bf8292ab (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251025 20:04:19.632766 31993 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:19.633597 31965 raft_consensus.cc:2749] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:19.633170 31801 consensus_queue.cc:237] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:19.638545 31970 consensus_meta.cc:287] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 1.037s user 0.011s sys 0.007s
W20251025 20:04:19.639254 31809 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.183s user 0.101s sys 0.174s
W20251025 20:04:19.639683 31809 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.211s user 0.101s sys 0.174s
W20251025 20:04:19.640101 32069 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.342s user 0.003s sys 0.037s
W20251025 20:04:19.640476 32069 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.711s user 0.004s sys 0.037s
I20251025 20:04:19.535061 32098 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:19.639153 31970 raft_consensus.cc:515] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.521462 32022 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.591908 31941 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.604853 32011 raft_consensus.cc:493] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.488845 31679 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.678627 32074 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.679467 32074 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.678617 32011 raft_consensus.cc:515] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.684322 32011 leader_election.cc:290] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.684927 32011 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.685580 31809 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:19.686440 31809 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.685531 32011 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:19.687489 32019 consensus_meta.cc:287] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.555s user 0.014s sys 0.029s
I20251025 20:04:19.688287 32019 consensus_queue.cc:1059] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:04:19.689013 31809 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:19.585273 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 1.68993e+06us
I20251025 20:04:19.690660 32011 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.692688 31679 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.693576 30660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6164729145ad47d6bbc2e7b3c6b49a22. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 20:04:19.698714 32076 log.cc:927] Time spent T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.252s user 0.000s sys 0.001s
I20251025 20:04:19.699307 31414 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:19.577208 32072 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
@ 0x7fd81ec5ed25 testing::TestSuite::Run() at ??:0
I20251025 20:04:19.605710 32073 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:19.729161 30670 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.020s sys 0.025s
I20251025 20:04:19.699373 32011 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.735775 32011 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.736845 30665 tablet_service.cc:1505] Processing CreateTablet for tablet 2d6dd59f914c41758a18803991215caf (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251025 20:04:19.737465 32095 tablet_bootstrap.cc:492] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:04:19.675318 32033 raft_consensus.cc:2749] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:19.698717 32062 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:19.734042 30670 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s user 0.020s sys 0.026s
I20251025 20:04:19.725728 31941 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.740979 32062 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:19.753306 32023 ts_tablet_manager.cc:1950] T e8c971a6d4854180b4d243fa79fbbc11 P 349d66b543bf47249ac78154b908b957: Deleting consensus metadata
I20251025 20:04:19.769265 32011 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.782217 32072 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:19.782822 30601 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.010s sys 0.030s
W20251025 20:04:19.787631 30601 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s user 0.010s sys 0.030s
I20251025 20:04:19.786213 30665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d6dd59f914c41758a18803991215caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:19.775372 32062 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:19.785648 30599 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.010s sys 0.038s
I20251025 20:04:19.802232 32084 raft_consensus.cc:493] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.802853 31941 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.803248 31809 raft_consensus.cc:3055] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:19.782790 32051 raft_consensus.cc:359] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.809468 31809 raft_consensus.cc:740] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:19.810781 31809 consensus_queue.cc:260] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.787735 32074 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.812635 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 514) took 2483 ms. Trace:
I20251025 20:04:19.809638 32051 raft_consensus.cc:385] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:19.812467 31809 raft_consensus.cc:3060] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
W20251025 20:04:19.813630 30599 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.837s user 0.010s sys 0.038s
I20251025 20:04:19.828998 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:19.829478 30746 raft_consensus.cc:1275] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 20:04:19.830228 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:19.813762 30662 tablet_service.cc:1505] Processing CreateTablet for tablet 5ee1ba6e04b74e1faf5c9819144cd740 (DEFAULT_TABLE table=test-workload [id=0622ca5c803d479d9858172423e767c7]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251025 20:04:19.831208 30747 raft_consensus.cc:2410] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:19.831559 30662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ee1ba6e04b74e1faf5c9819144cd740. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:19.831429 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.832094 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.834051 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.835836 32084 raft_consensus.cc:515] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.812759 31885 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.837805 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.838444 32023 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34862 (request call id 59) took 3425 ms. Trace:
I20251025 20:04:19.818996 32051 raft_consensus.cc:740] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
W20251025 20:04:19.848018 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 62353.4us
I20251025 20:04:19.848388 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:19.852541 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:19.854203 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.855120 30750 raft_consensus.cc:2393] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:19.856772 30744 raft_consensus.cc:1275] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:19.841202 31970 leader_election.cc:290] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:19.814316 30677 rpcz_store.cc:276] 1025 20:04:17.329479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.332481 (+ 3002us) service_pool.cc:225] Handling call
1025 20:04:17.332767 (+ 286us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:17.339517 (+ 6750us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:17.339545 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:17.340043 (+ 498us) log.cc:844] Serialized 26 byte log entry
1025 20:04:18.077961 (+737918us) spinlock_profiling.cc:243] Waited 4.02 ms on lock 0x7b18001fcde0. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8278d09a1 00007fd8278dd371 00007fd81ea702f7 00007fd827849219 00007fd81ea8cc4b 00007fd81ea8aeef 00007fd81ea8d1b2 00007fd8216d42bb 00007fd8216d9dc3 00007fd8216da171 00007fd821af565f
1025 20:04:19.621368 (+1543407us) raft_consensus.cc:1645] Marking committed up to 4
1025 20:04:19.641636 (+ 20268us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:19.668456 (+ 26820us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:19.812230 (+143774us) raft_consensus.cc:1685] finished
1025 20:04:19.812259 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:19.812540 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6497,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":25471,"spinlock_wait_cycles":9256704,"thread_start_us":1377882,"threads_started":1,"wal-append.queue_time_us":2454925}
I20251025 20:04:19.841302 32022 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.864430 30603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 810) took 2271 ms. Trace:
I20251025 20:04:19.871629 32084 leader_election.cc:290] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.838867 32023 rpcz_store.cc:276] 1025 20:04:16.432062 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.492879 (+ 60817us) service_pool.cc:225] Handling call
1025 20:04:16.705462 (+212583us) ts_tablet_manager.cc:1151] Deleting tablet e8c971a6d4854180b4d243fa79fbbc11
1025 20:04:16.705556 (+ 94us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 20:04:18.537418 (+1831862us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b58002e55d0. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821a95ffa 00007fd821b18097 00007fd821b41f15 00007fd825524afc 00007fd826b33daa 00007fd826b64a10 00007fd826b4afdd 00007fd826b4af89 00007fd826b4af19 00007fd826b4aee1
1025 20:04:18.562689 (+ 25271us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b58002e58d0. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821a95ffa 00007fd821b18097 00007fd821b41f15 00007fd825524afc 00007fd826b33daa 00007fd826b64a10 00007fd826b4afdd 00007fd826b4af89 00007fd826b4af19 00007fd826b4aee1
1025 20:04:19.083328 (+520639us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:19.520202 (+436874us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:19.838331 (+318129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":202156,"spinlock_wait_cycles":2486636160,"tablet-delete.queue_time_us":194222,"thread_start_us":115643,"threads_started":1}
I20251025 20:04:19.880755 30753 raft_consensus.cc:2468] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:19.869103 30603 rpcz_store.cc:276] 1025 20:04:17.587177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.587394 (+ 217us) service_pool.cc:225] Handling call
1025 20:04:17.587756 (+ 362us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:17.643850 (+ 56094us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:17.643969 (+ 119us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:17.644671 (+ 702us) log.cc:844] Serialized 26 byte log entry
1025 20:04:19.614481 (+1969810us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:19.614573 (+ 92us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:19.614683 (+ 110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:19.857992 (+243309us) raft_consensus.cc:1685] finished
1025 20:04:19.858018 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:19.858146 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29567,"raft.queue_time_us":63450,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":5672,"thread_start_us":1848683,"threads_started":1,"wal-append.queue_time_us":2130581}
I20251025 20:04:19.838462 31885 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.891712 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.896699 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.901403 30753 raft_consensus.cc:2393] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
W20251025 20:04:19.905500 30597 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.929s user 0.100s sys 0.211s
I20251025 20:04:19.905747 30746 raft_consensus.cc:2393] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 4: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
W20251025 20:04:19.906270 30597 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.930s user 0.100s sys 0.211s
W20251025 20:04:19.910346 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:19.912016 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:19.898095 32073 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.918934 32078 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:19.919844 32078 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.922830 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.901473 30749 raft_consensus.cc:1275] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251025 20:04:19.928540 30669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8afaad4899545a9a4169026b63c0699. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:19.928165 30597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 821) took 1976 ms. Trace:
I20251025 20:04:19.885787 32051 consensus_queue.cc:260] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.927328 32023 tablet_replica.cc:333] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: stopping tablet replica
I20251025 20:04:19.933024 30597 rpcz_store.cc:276] 1025 20:04:17.952193 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.953745 (+ 1552us) service_pool.cc:225] Handling call
1025 20:04:17.954274 (+ 529us) raft_consensus.cc:1471] Updating replica for 2 ops
1025 20:04:17.954637 (+ 363us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:17.954665 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1025 20:04:17.955219 (+ 554us) log.cc:844] Serialized 52 byte log entry
1025 20:04:18.972128 (+1016909us) raft_consensus.cc:1645] Marking committed up to 3
1025 20:04:19.907189 (+935061us) log.cc:844] Serialized 14 byte log entry
1025 20:04:19.916627 (+ 9438us) log.cc:844] Serialized 14 byte log entry
1025 20:04:19.916900 (+ 273us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:19.927818 (+ 10918us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:19.927892 (+ 74us) raft_consensus.cc:1685] finished
1025 20:04:19.927912 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:19.928094 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6346,"raft.queue_time_us":31916,"raft.run_cpu_time_us":466,"raft.run_wall_time_us":10136,"thread_start_us":1947359,"threads_started":2,"wal-append.queue_time_us":1344921}
I20251025 20:04:19.939131 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.930954 30736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 67) took 2833 ms. Trace:
I20251025 20:04:19.959177 30745 raft_consensus.cc:2368] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3.
I20251025 20:04:19.968103 31970 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:19.968760 31970 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:19.969688 32051 ts_tablet_manager.cc:1434] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.352s user 0.007s sys 0.005s
I20251025 20:04:19.947717 32044 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:19.984828 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:19.979547 30567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.017s sys 0.030s
I20251025 20:04:19.980851 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:19.985622 30567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.853s user 0.017s sys 0.030s
I20251025 20:04:19.986132 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.947080 32062 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:19.949802 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:04:19.986058 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 855017us
I20251025 20:04:19.989754 30567 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:04:19.987696 30600 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.017s sys 0.025s
W20251025 20:04:19.978638 30595 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.003s sys 0.036s
I20251025 20:04:19.984867 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:19.990887 30600 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s user 0.018s sys 0.026s
W20251025 20:04:19.991822 30595 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.860s user 0.004s sys 0.037s
I20251025 20:04:19.994290 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.000986 30745 raft_consensus.cc:2393] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 5: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:20.001010 32093 raft_consensus.cc:2804] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:20.002022 30748 raft_consensus.cc:1275] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:19.986662 30740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 71) took 2328 ms. Trace:
I20251025 20:04:20.010460 30753 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:20.012928 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:19.948285 30736 rpcz_store.cc:276] 1025 20:04:17.010705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.012835 (+ 2130us) service_pool.cc:225] Handling call
1025 20:04:18.091442 (+1078607us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:18.105315 (+ 13873us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:19.434743 (+1329428us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:19.787209 (+352466us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:19.844294 (+ 57085us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:20.015522 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.016489 30753 raft_consensus.cc:2393] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:20.016589 30595 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 814) took 2261 ms. Trace:
I20251025 20:04:20.015276 30668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8360e93e9b6d471791c2bd59bf8292ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 20:04:20.025647 30568 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.830s user 0.020s sys 0.019s
W20251025 20:04:20.026109 30568 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.883s user 0.020s sys 0.019s
I20251025 20:04:20.014595 32065 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:19.991914 32044 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.027215 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.032320 30746 raft_consensus.cc:2393] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 5: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:20.007962 32093 raft_consensus.cc:697] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:20.017244 30744 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:20.042338 30567 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.044997 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.046190 30745 raft_consensus.cc:2393] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
W20251025 20:04:20.047122 30596 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.009s sys 0.017s
W20251025 20:04:20.047439 30596 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.530s user 0.009s sys 0.017s
I20251025 20:04:20.022356 31970 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:20.041582 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.028306 31885 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.019315 30595 rpcz_store.cc:276] 1025 20:04:17.755277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.756711 (+ 1434us) service_pool.cc:225] Handling call
1025 20:04:17.757241 (+ 530us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:17.757718 (+ 477us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:17.757761 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:17.758433 (+ 672us) log.cc:844] Serialized 26 byte log entry
1025 20:04:19.131418 (+1372985us) raft_consensus.cc:1645] Marking committed up to 3
1025 20:04:19.999927 (+868509us) log.cc:844] Serialized 14 byte log entry
1025 20:04:20.005803 (+ 5876us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:20.016334 (+ 10531us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:20.016383 (+ 49us) raft_consensus.cc:1685] finished
1025 20:04:20.016396 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:20.016534 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21755,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":14832,"thread_start_us":2162428,"threads_started":2,"wal-append.queue_time_us":1730164}
I20251025 20:04:20.005409 30740 rpcz_store.cc:276] 1025 20:04:17.658718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.660602 (+ 1884us) service_pool.cc:225] Handling call
1025 20:04:18.158260 (+497658us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:18.253883 (+ 95623us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:19.479137 (+1225254us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:19.953636 (+474499us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:19.983082 (+ 29446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:20.066764 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.059055 30667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01d56eb384124dfcb1bcd35d77e89f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:04:20.062016 32100 tablet_replica.cc:333] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: stopping tablet replica
I20251025 20:04:20.062155 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:04:20.072144 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:20.063643 32051 tablet_bootstrap.cc:492] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:04:20.037989 32065 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:20.072791 32015 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.020s sys 0.006s
I20251025 20:04:20.084964 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.083633 30489 catalog_manager.cc:4985] TS 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): tablet e8c971a6d4854180b4d243fa79fbbc11 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) successfully deleted
I20251025 20:04:20.087090 30596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 843) took 1706 ms. Trace:
I20251025 20:04:20.066089 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.090927 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.113065 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.071571 31885 raft_consensus.cc:2804] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:20.131462 30601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 846) took 1112 ms. Trace:
I20251025 20:04:20.178568 30746 raft_consensus.cc:2468] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
W20251025 20:04:20.090592 32015 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.631s user 0.020s sys 0.006s
I20251025 20:04:20.099160 30752 raft_consensus.cc:3060] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:20.127139 30600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 848) took 1022 ms. Trace:
I20251025 20:04:20.188616 32065 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:20.147087 32101 log.cc:927] Time spent T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.085s user 0.001s sys 0.005s
W20251025 20:04:20.132215 32094 log.cc:927] Time spent T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.197s user 0.001s sys 0.001s
I20251025 20:04:20.137014 32095 tablet_bootstrap.cc:654] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
W20251025 20:04:20.097273 30644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.001s sys 0.018s
I20251025 20:04:20.185705 30750 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 6.
I20251025 20:04:20.129666 32044 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.133646 32022 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.203440 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:20.204558 30748 raft_consensus.cc:2393] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 4: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:20.205260 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.086601 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.146494 31965 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.150799 32078 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.133868 32070 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:20.128782 30645 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.192.195:46541 timed out after 1.508s (SENT) [suppressed 6 similar messages]
I20251025 20:04:20.134100 32108 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.186998 30600 rpcz_store.cc:276] 1025 20:04:19.087700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:19.090293 (+ 2593us) service_pool.cc:225] Handling call
1025 20:04:19.125687 (+ 35394us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:19.130398 (+ 4711us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:19.130448 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:19.131404 (+ 956us) log.cc:844] Serialized 26 byte log entry
1025 20:04:19.998103 (+866699us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:19.998167 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:19.998274 (+ 107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:20.109211 (+110937us) raft_consensus.cc:1685] finished
1025 20:04:20.109268 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:20.109498 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":866127,"threads_started":1,"wal-append.queue_time_us":968493}
W20251025 20:04:20.197698 31801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.015s sys 0.023s
I20251025 20:04:20.197332 31679 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:20.200453 30642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.007s sys 0.029s
W20251025 20:04:20.200258 30644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s user 0.001s sys 0.032s
I20251025 20:04:20.212309 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.174019 30596 rpcz_store.cc:276] 1025 20:04:18.380674 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:18.382877 (+ 2203us) service_pool.cc:225] Handling call
1025 20:04:18.406250 (+ 23373us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:18.406463 (+ 213us) raft_consensus.cc:1534] Early marking committed up to index 2
1025 20:04:18.418451 (+ 11988us) log.cc:844] Serialized 14 byte log entry
1025 20:04:20.048072 (+1629621us) log.cc:844] Serialized 14 byte log entry
1025 20:04:20.072089 (+ 24017us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:20.076353 (+ 4264us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:20.076445 (+ 92us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:20.076607 (+ 162us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:20.086994 (+ 10387us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11997,"prepare.queue_time_us":573799,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":9010,"raft.queue_time_us":20687,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":7396,"thread_start_us":1553554,"threads_started":2,"wal-append.queue_time_us":1307588}
W20251025 20:04:20.189877 30489 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 106296us
I20251025 20:04:20.218753 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:20.182122 30601 rpcz_store.cc:276] 1025 20:04:19.018791 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:19.022128 (+ 3337us) service_pool.cc:225] Handling call
1025 20:04:19.022662 (+ 534us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:19.023303 (+ 641us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:19.023339 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:19.024050 (+ 711us) log.cc:844] Serialized 26 byte log entry
1025 20:04:19.788049 (+763999us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:19.788092 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:19.788172 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:20.131174 (+343002us) raft_consensus.cc:1685] finished
1025 20:04:20.131199 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:20.131393 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":179,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":282,"spinlock_wait_cycles":51456,"thread_start_us":763438,"threads_started":1,"wal-append.queue_time_us":1010002}
I20251025 20:04:20.226961 30678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 549) took 1135 ms. Trace:
W20251025 20:04:20.225801 30643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.013s sys 0.033s
W20251025 20:04:20.239254 30643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s user 0.013s sys 0.033s
I20251025 20:04:20.202917 31751 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:20.218518 31801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s user 0.015s sys 0.023s
I20251025 20:04:20.243314 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:20.245265 30746 raft_consensus.cc:2393] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:20.237246 30678 rpcz_store.cc:276] 1025 20:04:19.091962 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:19.113493 (+ 21531us) service_pool.cc:225] Handling call
1025 20:04:19.133642 (+ 20149us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:19.602362 (+468720us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:19.602415 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:19.618929 (+ 16514us) log.cc:844] Serialized 26 byte log entry
1025 20:04:20.183767 (+564838us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:20.183808 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:20.183882 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:20.226510 (+ 42628us) raft_consensus.cc:1685] finished
1025 20:04:20.226552 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:20.226848 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":506044,"threads_started":1,"wal-append.queue_time_us":576484}
W20251025 20:04:20.236514 30642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.007s sys 0.033s
I20251025 20:04:20.253500 30752 raft_consensus.cc:2468] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:20.196776 31885 raft_consensus.cc:697] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:20.230401 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:20.259083 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 184236us
I20251025 20:04:20.242939 31679 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.150512 32093 consensus_queue.cc:237] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:20.266788 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 185269us
I20251025 20:04:20.266017 32044 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.269301 32044 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.271426 32044 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.220638 32070 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.274577 31885 consensus_queue.cc:237] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.275399 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:20.276144 30595 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:20.230515 32022 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.220790 31965 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.230664 32108 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.290544 32115 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:20.283133 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:20.291581 32025 raft_consensus.cc:2243] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:04:20.294852 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:20.310937 30596 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:20.293768 30719 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:20.294127 30601 raft_consensus.cc:1240] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:20.313745 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.314489 30673 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:20.278136 32111 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:20.315199 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 250712us
I20251025 20:04:20.293306 30599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 808) took 2710 ms. Trace:
I20251025 20:04:20.313148 30598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 837) took 2042 ms. Trace:
I20251025 20:04:20.283799 32070 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:20.285686 30738 consensus_meta.cc:287] T 5ee1ba6e04b74e1faf5c9819144cd740 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.671s user 0.004s sys 0.012s
I20251025 20:04:20.258570 31751 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.285077 32044 raft_consensus.cc:493] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:20.327574 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.327752 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.328336 30673 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:20.308789 32073 raft_consensus.cc:493] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:20.330265 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:20.331059 30679 raft_consensus.cc:3060] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Advancing to term 3
I20251025 20:04:20.332688 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:20.336172 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 56589.1us
I20251025 20:04:20.312891 32072 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.319465 30598 rpcz_store.cc:276] 1025 20:04:18.271006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:18.271487 (+ 481us) service_pool.cc:225] Handling call
1025 20:04:18.271884 (+ 397us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:18.272300 (+ 416us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:18.272342 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:18.298288 (+ 25946us) log.cc:844] Serialized 26 byte log entry
1025 20:04:18.547086 (+248798us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b48000027d8. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd826a47971 00007fd826a3ea37 00007fd826a3e8ca 00007fd826b35a0f 00007fd826b356d7 00007fd826b49cfc 00007fd826b49ca8 00007fd826b49c28 00007fd826b49be3 00007fd826b48e8c 00007fd825a86326
1025 20:04:19.587278 (+1040192us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:19.617849 (+ 30571us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:19.637101 (+ 19252us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:20.293017 (+655916us) raft_consensus.cc:1685] finished
1025 20:04:20.293051 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:20.313063 (+ 20012us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":346,"raft.run_cpu_time_us":1402,"raft.run_wall_time_us":273941,"spinlock_wait_cycles":30393088,"thread_start_us":1099124,"threads_started":1,"wal-append.queue_time_us":1568841}
I20251025 20:04:20.319831 32011 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.329751 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:20.318207 30599 rpcz_store.cc:276] 1025 20:04:17.583039 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.604435 (+ 21396us) service_pool.cc:225] Handling call
1025 20:04:17.604905 (+ 470us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:17.662091 (+ 57186us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:17.663943 (+ 1852us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:18.689381 (+1025438us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b2c000014c8. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8204399b1 00007fd820448fe2 00007fd821b75701 00007fd821b39fda 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22
1025 20:04:18.916451 (+227070us) log.cc:844] Serialized 26 byte log entry
1025 20:04:19.853741 (+937290us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:19.861739 (+ 7998us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:19.882901 (+ 21162us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:20.146627 (+263726us) spinlock_profiling.cc:243] Waited 7.96 ms on lock 0x7b240009db68. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e8e25a4 00007fd81e8e06ab 00007fd81e96f409 00007fd81e96f095 00007fd81e96e6dd 00007fd8216be20e 00007fd8216d593c 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1
1025 20:04:20.278307 (+131680us) raft_consensus.cc:1685] finished
1025 20:04:20.279825 (+ 1518us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:20.285824 (+ 5999us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47667,"raft.queue_time_us":49958,"raft.run_cpu_time_us":585,"raft.run_wall_time_us":51068,"spinlock_wait_cycles":25539200,"thread_start_us":872127,"threads_started":1,"wal-append.queue_time_us":1063707}
I20251025 20:04:20.348067 30675 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:20.351500 32112 raft_consensus.cc:493] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.352517 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.337128 32062 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.358584 32084 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.359198 32084 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.359993 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.360697 30602 raft_consensus.cc:2468] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:20.346962 32025 raft_consensus.cc:2272] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:04:20.355847 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.364125 30678 raft_consensus.cc:2468] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:20.370873 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.371603 31965 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.370908 32022 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.372416 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.373246 32108 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.315140 32115 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.373729 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.374480 30604 raft_consensus.cc:2468] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:20.375690 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.376019 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.388916 30673 raft_consensus.cc:2393] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 4: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:20.335942 32073 raft_consensus.cc:515] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.376574 30600 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:20.390015 30675 raft_consensus.cc:2468] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:20.322135 32111 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.361677 32084 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.346458 32044 raft_consensus.cc:515] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.400461 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:20.406136 30599 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:20.363322 32011 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.350615 32072 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.410176 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.411713 30570 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:20.412392 32011 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.414556 32070 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.390035 31965 raft_consensus.cc:493] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.417708 30673 raft_consensus.cc:2393] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 2: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:20.418080 30679 raft_consensus.cc:2468] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:20.416113 30672 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:20.399372 32078 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.426036 30601 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:20.430266 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.434427 30744 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:20.435745 32072 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.437480 30753 raft_consensus.cc:1240] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
W20251025 20:04:20.438271 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:20.419572 30738 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53746 (request call id 73) took 2707 ms. Trace:
I20251025 20:04:20.401432 32062 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:20.442484 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:20.443586 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.421670 31679 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.417711 32022 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.437212 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.469214 30673 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:20.438396 32126 raft_consensus.cc:2749] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:20.439769 31414 consensus_queue.cc:1059] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
W20251025 20:04:20.461479 32103 log.cc:927] Time spent T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.246s user 0.001s sys 0.000s
I20251025 20:04:20.461838 30596 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
W20251025 20:04:20.434160 31809 consensus_meta.cc:287] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.592s user 0.012s sys 0.002s
I20251025 20:04:20.467463 30595 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:20.415064 32070 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.419355 32015 consensus_queue.cc:1059] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251025 20:04:20.437794 32124 tablet_bootstrap.cc:492] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:04:20.401592 32112 raft_consensus.cc:515] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.470582 30645 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:20.427481 31965 raft_consensus.cc:515] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.443468 32121 raft_consensus.cc:2764] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251025 20:04:20.427886 32078 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.473071 31801 raft_consensus.cc:493] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:20.488078 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.488831 30673 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:20.489271 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:20.489980 30595 raft_consensus.cc:3060] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:20.493358 32115 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.493512 32070 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.495416 31965 leader_election.cc:290] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.496505 32111 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.498486 32096 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:20.499296 32112 leader_election.cc:290] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.499138 32096 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.500708 32073 leader_election.cc:290] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.482981 32022 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:20.503194 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:20.507148 30670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41088 (request call id 822) took 3150 ms. Trace:
I20251025 20:04:20.507719 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.508396 30679 raft_consensus.cc:2468] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:20.508611 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.481994 31751 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.516674 32100 raft_consensus.cc:2243] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 20:04:20.518265 32062 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.446399 30738 rpcz_store.cc:276] 1025 20:04:17.708982 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.715071 (+ 6089us) service_pool.cc:225] Handling call
1025 20:04:18.256355 (+541284us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:18.270119 (+ 13764us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:19.613696 (+1343577us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:20.343166 (+729470us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:20.416535 (+ 73369us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:20.522164 32025 ts_tablet_manager.cc:1916] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:04:20.524101 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.525408 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.532222 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.532999 30674 raft_consensus.cc:2468] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 1.
I20251025 20:04:20.534376 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.535130 30671 raft_consensus.cc:2468] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:20.521579 32078 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.535466 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.541240 30672 raft_consensus.cc:1275] T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:20.536392 30675 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 7.
I20251025 20:04:20.536881 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.507553 30670 rpcz_store.cc:276] 1025 20:04:17.356467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.662385 (+305918us) service_pool.cc:225] Handling call
1025 20:04:17.662827 (+ 442us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:18.978829 (+1316002us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:18.978875 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:18.979667 (+ 792us) log.cc:844] Serialized 26 byte log entry
1025 20:04:19.740926 (+761259us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:19.742453 (+ 1527us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:19.760058 (+ 17605us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:20.460488 (+700430us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b440002304c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81e99c905 00007fd81e99d322 00007fd8216e6231 00007fd8216e1b5a 00007fd8216d591e 00007fd8216d4c66 00007fd8216d4888 00007fd8216de2c1 00007fd8216de279 00007fd8216de209
1025 20:04:20.503487 (+ 42999us) raft_consensus.cc:1685] finished
1025 20:04:20.503509 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:20.507076 (+ 3567us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":113571,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":17148,"spinlock_wait_cycles":6464384,"thread_start_us":2075686,"threads_started":2,"wal-append.queue_time_us":1112069}
I20251025 20:04:20.571156 30601 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:20.571382 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.502550 32044 leader_election.cc:290] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.501744 32096 leader_election.cc:290] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:20.544914 32051 tablet_bootstrap.cc:654] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:20.570838 30673 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:20.583516 30677 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:20.503159 32022 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.585214 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.594578 30670 raft_consensus.cc:2468] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:20.535533 30595 raft_consensus.cc:2468] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:20.590386 30567 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:04:20.594480 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 125602us
I20251025 20:04:20.586937 30678 raft_consensus.cc:2410] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:20.539152 31809 raft_consensus.cc:3055] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:20.514751 32115 raft_consensus.cc:2749] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:20.605082 30567 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.539480 31414 raft_consensus.cc:3055] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:20.503499 31801 raft_consensus.cc:515] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:20.601990 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 131550us
I20251025 20:04:20.596977 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:20.571991 32107 raft_consensus.cc:493] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:20.609221 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.609300 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.626607 30671 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:20.613759 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.627779 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.545786 32019 consensus_queue.cc:1059] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20251025 20:04:20.628466 30675 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:20.629700 30673 raft_consensus.cc:2393] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:20.630206 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.634852 30670 raft_consensus.cc:2393] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:20.606643 32100 raft_consensus.cc:2272] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:04:20.638175 30603 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:20.637080 30567 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.602806 31809 raft_consensus.cc:740] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:20.606240 32121 raft_consensus.cc:493] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:20.611249 32044 raft_consensus.cc:2804] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:20.642204 30567 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.608453 31414 raft_consensus.cc:740] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:20.611332 32022 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:20.647233 30645 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters: 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:20.646965 30570 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:20.646687 32135 log.cc:927] Time spent T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
I20251025 20:04:20.659314 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:20.671600 30670 raft_consensus.cc:2393] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:20.674631 30668 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 72) took 1064 ms. Trace:
I20251025 20:04:20.646119 31679 raft_consensus.cc:2804] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:20.654873 32122 raft_consensus.cc:2804] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:20.678110 30567 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.645524 32044 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.654232 32022 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.692343 32022 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:20.677801 31679 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:20.695029 31679 raft_consensus.cc:3060] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:20.698236 30664 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 68) took 1222 ms. Trace:
I20251025 20:04:20.698715 30664 rpcz_store.cc:276] 1025 20:04:19.475921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:19.476090 (+ 169us) service_pool.cc:225] Handling call
1025 20:04:19.479629 (+ 3539us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:19.479840 (+ 211us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:20.144322 (+664482us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:20.644349 (+500027us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:20.698155 (+ 53806us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:20.656395 32121 raft_consensus.cc:515] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.675148 30668 rpcz_store.cc:276] 1025 20:04:19.608131 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:19.616289 (+ 8158us) service_pool.cc:225] Handling call
1025 20:04:19.631586 (+ 15297us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:19.631731 (+ 145us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:20.269226 (+637495us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:20.629469 (+360243us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:20.669767 (+ 40298us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 20:04:20.687124 30716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:20.687337 32100 ts_tablet_manager.cc:1916] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:04:20.687867 31809 consensus_queue.cc:260] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.673381 32115 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:20.691847 32095 tablet_bootstrap.cc:492] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
I20251025 20:04:20.680537 32108 raft_consensus.cc:2749] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251025 20:04:20.630748 32107 raft_consensus.cc:515] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:20.707087 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 69322.5us
I20251025 20:04:20.703778 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:20.710397 32107 leader_election.cc:290] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.713537 32124 tablet_bootstrap.cc:654] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:20.715444 30678 raft_consensus.cc:2393] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:20.715763 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.716331 30670 raft_consensus.cc:2468] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:20.706226 32115 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.720090 32115 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:20.724046 30661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 67) took 1248 ms. Trace:
I20251025 20:04:20.724509 30661 rpcz_store.cc:276] 1025 20:04:19.475507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:19.478813 (+ 3306us) service_pool.cc:225] Handling call
1025 20:04:19.480988 (+ 2175us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:19.481118 (+ 130us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:20.489261 (+1008143us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:20.723496 (+234235us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:20.723976 (+ 480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:20.727140 30671 raft_consensus.cc:3060] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:20.704541 32122 raft_consensus.cc:697] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:20.729593 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.729827 32122 consensus_queue.cc:237] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.730481 30675 raft_consensus.cc:2393] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 4: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
W20251025 20:04:20.732344 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 54335.8us
I20251025 20:04:20.744602 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.745208 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.745374 30673 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:20.747227 31801 leader_election.cc:290] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.748005 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:20.748745 30674 raft_consensus.cc:2393] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:20.748356 32115 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.749207 30660 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41086 (request call id 71) took 1272 ms. Trace:
I20251025 20:04:20.702891 32044 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:20.751654 30670 raft_consensus.cc:1240] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:20.743638 32095 ts_tablet_manager.cc:1403] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 1.011s user 0.012s sys 0.028s
I20251025 20:04:20.749671 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:20.754732 30676 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 4
I20251025 20:04:20.756131 30567 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:04:20.763801 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 116878us
I20251025 20:04:20.775599 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:20.812736 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 56718.5us
I20251025 20:04:20.815925 30567 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.816330 30501 catalog_manager.cc:5649] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 4, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 4 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:20.774349 32120 raft_consensus.cc:2804] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:20.749637 30660 rpcz_store.cc:276] 1025 20:04:19.477250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:19.531522 (+ 54272us) service_pool.cc:225] Handling call
1025 20:04:19.533819 (+ 2297us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:19.577918 (+ 44099us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:20.345644 (+767726us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:20.748602 (+402958us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:20.749138 (+ 536us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:20.818171 31801 raft_consensus.cc:2804] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:20.776597 30645 leader_election.cc:400] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:20.824865 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.839015 30645 leader_election.cc:403] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20251025 20:04:20.839282 30670 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:20.738318 31414 consensus_queue.cc:260] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.827989 30676 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:20.829437 32121 leader_election.cc:290] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:20.830358 30567 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.830408 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.849877 30672 raft_consensus.cc:2468] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:20.831025 31965 raft_consensus.cc:2804] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:20.831377 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:20.832239 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.814482 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:20.840916 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 214139us
I20251025 20:04:20.841497 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.842741 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.844571 31414 raft_consensus.cc:3060] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:20.847385 32025 ts_tablet_manager.cc:1929] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251025 20:04:20.847826 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.853562 30678 raft_consensus.cc:3060] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:20.852064 30673 raft_consensus.cc:3060] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:20.859858 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.841210 31801 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:20.869191 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 135853us
I20251025 20:04:20.861915 30567 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.861713 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.862480 30719 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:20.864003 30671 raft_consensus.cc:2410] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251025 20:04:20.864589 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:20.859993 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:20.861863 32108 raft_consensus.cc:2804] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:20.851923 31965 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.871865 30674 raft_consensus.cc:3060] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:20.872207 30570 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:20.851406 32120 raft_consensus.cc:697] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:20.843704 32115 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:20.877948 30673 raft_consensus.cc:2468] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:20.862471 32025 log.cc:1199] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: Deleting WAL directory at /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-2-root/wals/fea0c74c9c0c4a69afe072c45c4cbe99
I20251025 20:04:20.884012 32025 ts_tablet_manager.cc:1950] T fea0c74c9c0c4a69afe072c45c4cbe99 P 783f8fd6d80141b7912f66a954a6f92e: Deleting consensus metadata
I20251025 20:04:20.873212 31801 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
W20251025 20:04:20.886907 32141 log.cc:927] Time spent T 20fe84e124304a7a9d92910cfa619beb P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20251025 20:04:20.888142 30645 leader_election.cc:400] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3.
I20251025 20:04:20.888481 30645 leader_election.cc:403] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:04:20.889146 32073 consensus_queue.cc:1048] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:04:20.890679 32131 raft_consensus.cc:2749] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717) with higher term. Message: Invalid argument: T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:20.882537 32062 consensus_queue.cc:1048] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20251025 20:04:20.895128 31751 raft_consensus.cc:2804] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:20.895604 30570 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:20.895848 31751 raft_consensus.cc:697] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:20.890807 32073 raft_consensus.cc:2749] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3.
I20251025 20:04:20.880939 30567 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.897971 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:20.899857 30676 raft_consensus.cc:2468] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:20.889763 30645 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:20.878415 31965 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:20.903682 30567 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.874927 32108 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.882690 31885 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:20.901201 32132 consensus_queue.cc:1048] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20251025 20:04:20.904698 32126 raft_consensus.cc:2749] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:20.916095 30719 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:20.910270 32062 raft_consensus.cc:2804] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:20.906540 32108 raft_consensus.cc:3060] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:20.923713 30719 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:20.907742 32093 raft_consensus.cc:2804] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:04:20.900905 31751 consensus_queue.cc:237] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.943169 32022 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:20.927644 32070 consensus_queue.cc:1048] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20251025 20:04:20.934213 32019 raft_consensus.cc:2804] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:20.935513 32025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53746 (request call id 60) took 4414 ms. Trace:
I20251025 20:04:20.936190 30719 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:20.919785 32078 raft_consensus.cc:2804] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:20.939148 32051 tablet_bootstrap.cc:492] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:04:20.939627 32146 raft_consensus.cc:2804] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:20.928783 30567 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:20.907727 31885 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.928181 32093 raft_consensus.cc:493] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:20.962297 30674 raft_consensus.cc:2468] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:20.967859 31809 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:20.991500 30645 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:20.995612 30719 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:20.945809 32062 raft_consensus.cc:697] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:20.976051 32019 raft_consensus.cc:493] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:20.981576 32023 raft_consensus.cc:2243] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Raft consensus shutting down.
W20251025 20:04:21.002427 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 73718.7us
I20251025 20:04:21.007758 30567 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:21.006294 30719 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:21.006318 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 110867us
I20251025 20:04:21.009280 30570 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.003361 30645 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:20.988757 32144 raft_consensus.cc:2804] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:20.989255 32146 raft_consensus.cc:493] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:21.000679 32051 ts_tablet_manager.cc:1403] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 0.966s user 0.016s sys 0.016s
I20251025 20:04:20.999063 32093 raft_consensus.cc:3060] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:20.992949 31885 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:21.001331 32115 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251025 20:04:21.007066 32132 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251025 20:04:21.008293 32019 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:21.023079 30501 catalog_manager.cc:5649] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 3 to 4, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:21.047793 32124 tablet_bootstrap.cc:492] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
W20251025 20:04:21.048554 32147 log.cc:927] Time spent T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251025 20:04:21.053541 31414 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:21.052963 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.015870 32144 raft_consensus.cc:493] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:21.051487 30570 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:21.054713 31414 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.054888 30751 raft_consensus.cc:2468] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:04:21.020967 32111 raft_consensus.cc:2804] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:21.050071 32070 raft_consensus.cc:2764] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251025 20:04:21.021842 32146 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:21.059156 30719 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:21.028338 32115 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:21.025626 32140 raft_consensus.cc:2804] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:21.070149 31414 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:21.069016 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d33d554021204714a7a71332612e2070" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:21.058502 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:04:21.041846 32132 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:21.056288 32144 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:21.078951 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.076913 31414 raft_consensus.cc:2804] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:21.049772 32149 raft_consensus.cc:2804] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:21.060673 32124 ts_tablet_manager.cc:1403] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.654s user 0.016s sys 0.040s
I20251025 20:04:21.056499 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.020787 32095 raft_consensus.cc:359] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.062116 31970 raft_consensus.cc:2804] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:21.082386 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.024439 31679 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.084705 32132 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Advancing to term 8
I20251025 20:04:21.075915 32140 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:21.077054 32111 raft_consensus.cc:697] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:21.086345 32099 raft_consensus.cc:3055] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:21.087864 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.090654 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:21.091055 31414 raft_consensus.cc:697] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:20.974874 32025 rpcz_store.cc:276] 1025 20:04:16.521323 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.523576 (+ 2253us) service_pool.cc:225] Handling call
1025 20:04:18.633219 (+2109643us) ts_tablet_manager.cc:1151] Deleting tablet fea0c74c9c0c4a69afe072c45c4cbe99
1025 20:04:18.633296 (+ 77us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 20:04:20.733293 (+2099997us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:20.847041 (+113748us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:20.935403 (+ 88362us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1604532,"tablet-delete.queue_time_us":2108511}
I20251025 20:04:21.091084 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.093124 30719 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:21.014146 32126 consensus_queue.cc:1048] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.058s
I20251025 20:04:21.055392 32044 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.074415 32115 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Advancing to term 7
W20251025 20:04:21.098001 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 46606.5us
I20251025 20:04:21.085412 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.125876 30570 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:21.105768 32158 raft_consensus.cc:2749] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:21.084489 32149 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:21.115691 30645 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:21.116550 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:21.117681 32078 raft_consensus.cc:2810] Couldn't start leader election after successful pre-election: Illegal state: RaftConsensus is not running: State = Stopping
I20251025 20:04:21.095953 32095 raft_consensus.cc:385] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:21.138947 30676 raft_consensus.cc:1240] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 5. Current term is 6. Ops: []
I20251025 20:04:21.141438 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:21.144006 30678 raft_consensus.cc:2468] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
W20251025 20:04:21.147145 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 64567.5us
I20251025 20:04:21.151193 30719 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:21.151664 32082 raft_consensus.cc:2804] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:21.152242 32082 raft_consensus.cc:697] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:21.153107 32152 raft_consensus.cc:493] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:21.154096 32121 raft_consensus.cc:2804] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:21.154482 32121 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:21.154872 32111 consensus_queue.cc:237] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.123032 32099 raft_consensus.cc:740] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:21.159381 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:21.107122 32140 raft_consensus.cc:3060] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:21.161280 31414 consensus_queue.cc:237] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.097018 31801 raft_consensus.cc:515] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.140331 31970 raft_consensus.cc:697] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:21.154827 32121 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:21.099809 32108 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:21.180344 30645 leader_election.cc:343] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Not found: Tablet not found: e8c971a6d4854180b4d243fa79fbbc11
I20251025 20:04:21.141971 32073 raft_consensus.cc:493] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:21.149747 32131 raft_consensus.cc:2804] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:21.091493 32062 consensus_queue.cc:237] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.148821 32095 raft_consensus.cc:740] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:04:21.146472 32149 raft_consensus.cc:3060] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:21.157341 31993 raft_consensus.cc:2804] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:21.146086 32022 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.211087 30604 raft_consensus.cc:1240] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:21.180160 30501 catalog_manager.cc:5649] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 3 to 4, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 4 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:21.189242 30567 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:21.190871 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:21.220099 30598 raft_consensus.cc:1240] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:21.222074 30673 raft_consensus.cc:1275] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251025 20:04:21.223948 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7294f698d29e4c919745b460e4e25eeb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:21.224428 32070 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:21.224970 32070 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:21.225512 32070 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Advancing to term 6
W20251025 20:04:21.226806 32154 log.cc:927] Time spent T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20251025 20:04:21.226663 32124 raft_consensus.cc:359] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.227531 32124 raft_consensus.cc:385] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:21.227849 32124 raft_consensus.cc:740] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:04:21.228186 32108 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.228116 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:21.229238 30675 raft_consensus.cc:2410] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:21.228611 32124 consensus_queue.cc:260] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.217226 32151 raft_consensus.cc:493] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:21.231786 32155 log.cc:927] Time spent T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20251025 20:04:21.232417 32025 tablet_replica.cc:333] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: stopping tablet replica
I20251025 20:04:21.179317 32078 raft_consensus.cc:2749] T 5f27e13dcf894a1487ed057731df6174 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:21.237349 30567 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
W20251025 20:04:21.185509 30630 tablet.cc:2378] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:04:21.241930 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:21.187649 32131 raft_consensus.cc:493] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
W20251025 20:04:21.225534 32150 log.cc:927] Time spent T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
W20251025 20:04:21.296775 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 106609us
W20251025 20:04:21.316186 30716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:21.324088 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:21.294286 32131 raft_consensus.cc:3060] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:21.211117 32073 raft_consensus.cc:515] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.348001 32157 raft_consensus.cc:2804] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:21.348731 32157 raft_consensus.cc:493] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:21.349274 32157 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:21.349562 31679 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.354468 31885 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.319111 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.362511 32124 ts_tablet_manager.cc:1434] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.268s user 0.009s sys 0.008s
I20251025 20:04:21.363034 31970 consensus_queue.cc:237] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:21.365820 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 48112.4us
W20251025 20:04:21.408509 30567 leader_election.cc:343] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:04:21.411464 30596 raft_consensus.cc:1240] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 5. Current term is 6. Ops: []
W20251025 20:04:21.417534 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 88167.5us
I20251025 20:04:21.296600 32095 consensus_queue.cc:260] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.437317 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.439479 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:21.409474 31993 raft_consensus.cc:697] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:21.416950 32023 raft_consensus.cc:2272] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:04:21.441650 30716 leader_election.cc:304] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.451742 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:21.451996 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:21.453367 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:21.454245 32153 log.cc:927] Time spent T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.296s user 0.001s sys 0.000s
I20251025 20:04:21.455022 32095 ts_tablet_manager.cc:1434] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.700s user 0.007s sys 0.005s
W20251025 20:04:21.455375 32095 ts_tablet_manager.cc:1467] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e: Tablet startup took 700ms
I20251025 20:04:21.446630 31414 raft_consensus.cc:493] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:21.414315 31679 consensus_queue.cc:1048] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20251025 20:04:21.422920 32044 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:21.462257 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 336498us
I20251025 20:04:21.445077 30489 catalog_manager.cc:4985] TS 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): tablet fea0c74c9c0c4a69afe072c45c4cbe99 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) successfully deleted
I20251025 20:04:21.468230 31414 raft_consensus.cc:515] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.153635 32152 raft_consensus.cc:515] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.456327 31801 leader_election.cc:290] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.470057 32124 tablet_bootstrap.cc:492] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:04:21.004774 32120 consensus_queue.cc:237] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.495841 30570 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:21.498479 31993 consensus_queue.cc:237] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:21.500012 32162 log.cc:927] Time spent T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
W20251025 20:04:21.502183 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 95111.6us
I20251025 20:04:21.503460 31885 leader_election.cc:290] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.231432 32151 raft_consensus.cc:515] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.323290 32082 consensus_queue.cc:237] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.506258 32151 leader_election.cc:290] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.508302 32119 consensus_queue.cc:1059] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251025 20:04:21.488615 30489 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 47939.2us
I20251025 20:04:21.510895 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:21.511855 30598 raft_consensus.cc:2393] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:21.514154 32168 raft_consensus.cc:3055] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Stepping down as leader of term 2
W20251025 20:04:21.499269 32165 log.cc:927] Time spent T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20251025 20:04:21.518505 30677 raft_consensus.cc:3060] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:21.523864 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "380df7888a29498d99ec3fe64410a542" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:21.508704 30595 raft_consensus.cc:1240] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 4. Current term is 5. Ops: []
W20251025 20:04:21.493095 31965 consensus_meta.cc:287] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.582s user 0.021s sys 0.005s
I20251025 20:04:21.514729 32168 raft_consensus.cc:740] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:21.233165 32099 consensus_queue.cc:260] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.491096 32096 raft_consensus.cc:2749] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:21.539212 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:21.542749 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:21.541649 32111 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:21.551425 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.551549 32111 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:21.462428 32095 ts_tablet_manager.cc:1469] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e: Trace:
1025 20:04:17.011145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.011380 (+ 235us) service_pool.cc:225] Handling call
1025 20:04:17.033108 (+ 21728us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:17.056537 (+ 23429us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:17.996867 (+940330us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:18.141786 (+144919us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:19.218990 (+1077204us) inbound_call.cc:173] Queueing success response
1025 20:04:19.715983 (+496993us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1025 20:04:20.754574 (+1038591us) ts_tablet_manager.cc:1435] Starting tablet replica
1025 20:04:20.819421 (+ 64847us) tablet_replica.cc:261] Starting instrumentation
1025 20:04:20.820172 (+ 751us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":41367,"raft.queue_time_us":32430,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":33125,"tablet-open.queue_time_us":1354276,"thread_start_us":847867,"threads_started":1}
I20251025 20:04:21.553601 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.551579 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.557665 32111 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.558497 32051 raft_consensus.cc:359] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:21.515558 32072 raft_consensus.cc:493] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:21.517705 32022 raft_consensus.cc:2749] T d4af0795c8ff48d9b639058ea7111f12 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:21.565646 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:21.572489 30753 raft_consensus.cc:2393] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:21.566468 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:21.568845 30604 raft_consensus.cc:1240] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:21.609460 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.500012 32070 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.563617 32051 raft_consensus.cc:385] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:21.614595 30750 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:21.634743 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.635895 30749 raft_consensus.cc:2468] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:21.639127 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.640767 32168 consensus_queue.cc:260] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.545595 31965 raft_consensus.cc:515] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.676532 30598 raft_consensus.cc:1240] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 5. Current term is 6. Ops: []
W20251025 20:04:21.680373 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 119194us
I20251025 20:04:21.682518 30645 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:21.617286 32051 raft_consensus.cc:740] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
W20251025 20:04:21.691387 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 187261us
I20251025 20:04:21.691771 31414 leader_election.cc:290] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:21.692276 30716 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.693039 31414 raft_consensus.cc:2749] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:21.694401 30716 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.695465 31414 raft_consensus.cc:2749] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:21.696194 32100 ts_tablet_manager.cc:1929] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 20:04:21.696981 32073 leader_election.cc:290] T 827742f9e2064e96a4fdcbd6a9b9a4a2 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.698163 30716 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.698935 32100 log.cc:1199] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: Deleting WAL directory at /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-1-root/wals/e8c971a6d4854180b4d243fa79fbbc11
I20251025 20:04:21.699784 32173 raft_consensus.cc:2749] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:21.700276 32100 ts_tablet_manager.cc:1950] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: Deleting consensus metadata
I20251025 20:04:21.702899 30716 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.632378 32163 raft_consensus.cc:493] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
@ 0x7fd81ec73815 testing::internal::UnitTestImpl::RunAllTests() at ??:0
I20251025 20:04:21.707547 30750 raft_consensus.cc:1275] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 2. (index mismatch)
I20251025 20:04:21.709704 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.710989 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.608496 32072 raft_consensus.cc:515] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.719796 32095 tablet_bootstrap.cc:492] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:04:21.714607 32183 raft_consensus.cc:2749] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251025 20:04:21.706387 32168 raft_consensus.cc:3060] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:21.739708 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.739611 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.740329 30716 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.722625 32108 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:21.729609 30747 raft_consensus.cc:2468] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:21.749066 30716 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.754685 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.755764 31965 leader_election.cc:290] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.756220 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.760317 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:21.760684 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:21.749581 30719 consensus_peers.cc:597] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e -> Peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Couldn't send request to peer 349d66b543bf47249ac78154b908b957. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
W20251025 20:04:21.765154 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 62706.5us
I20251025 20:04:21.749248 32015 raft_consensus.cc:2749] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:21.768653 30716 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.787451 32099 raft_consensus.cc:3060] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:21.735719 32051 consensus_queue.cc:260] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:21.790045 32051 ts_tablet_manager.cc:1434] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.765s user 0.012s sys 0.000s
W20251025 20:04:21.790478 32051 ts_tablet_manager.cc:1467] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957: Tablet startup took 769ms
I20251025 20:04:21.790812 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.791074 32179 raft_consensus.cc:2804] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:21.791718 32179 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:21.792205 32179 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:21.766630 32131 raft_consensus.cc:515] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.796372 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:21.796579 32131 leader_election.cc:290] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.799808 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.800625 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.803750 32023 ts_tablet_manager.cc:1916] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:04:21.813308 30671 raft_consensus.cc:1275] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:21.726104 32163 raft_consensus.cc:515] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:21.814128 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 45519.4us
I20251025 20:04:21.814658 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:21.820169 30672 raft_consensus.cc:2393] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 6: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:21.769488 32124 tablet_bootstrap.cc:654] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:21.767105 32152 leader_election.cc:290] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.825395 30675 raft_consensus.cc:1275] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251025 20:04:21.746446 32108 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:21.781755 31809 consensus_meta.cc:287] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.784s user 0.012s sys 0.000s
I20251025 20:04:21.822130 30716 leader_election.cc:304] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.825835 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
W20251025 20:04:21.818636 32172 log.cc:927] Time spent T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.127s user 0.003s sys 0.000s
I20251025 20:04:21.832846 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7294f698d29e4c919745b460e4e25eeb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.836797 30716 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:21.839530 30753 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:21.839915 32082 raft_consensus.cc:2749] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:21.822808 32065 raft_consensus.cc:2749] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251025 20:04:21.840288 32178 log.cc:927] Time spent T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20251025 20:04:21.840991 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:21.841229 31970 consensus_queue.cc:1059] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:04:21.842075 32119 consensus_queue.cc:1059] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20251025 20:04:21.842489 32082 raft_consensus.cc:3055] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Stepping down as leader of term 5
I20251025 20:04:21.842919 32082 raft_consensus.cc:740] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:21.843083 32119 raft_consensus.cc:3055] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:21.843456 32119 raft_consensus.cc:740] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:21.843812 32082 consensus_queue.cc:260] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.844235 32119 consensus_queue.cc:260] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.845191 32082 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:21.845539 32119 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
W20251025 20:04:21.823781 32181 log.cc:927] Time spent T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.150s user 0.002s sys 0.000s
I20251025 20:04:21.837450 30750 raft_consensus.cc:2468] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 0.
I20251025 20:04:21.824034 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.859535 30677 raft_consensus.cc:1275] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:21.819955 32100 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41086 (request call id 59) took 3389 ms. Trace:
W20251025 20:04:21.845798 32182 log.cc:927] Time spent T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
W20251025 20:04:21.846433 32176 log.cc:927] Time spent T 7904fa0f61dd4c9fadc39faddc8f908c P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.250s user 0.001s sys 0.001s
I20251025 20:04:21.820031 31965 consensus_queue.cc:1048] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20251025 20:04:21.835400 32131 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:21.871326 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.871994 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.878026 32025 raft_consensus.cc:2243] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Raft consensus shutting down.
I20251025 20:04:21.877976 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:21.878559 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:21.879117 30595 raft_consensus.cc:3060] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:21.862046 32098 raft_consensus.cc:2804] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:21.879732 32025 raft_consensus.cc:2272] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:04:21.880122 32098 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:21.880553 32098 raft_consensus.cc:3060] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:21.879189 30749 raft_consensus.cc:3060] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:21.855085 32070 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.875202 30501 catalog_manager.cc:5649] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 4 to 7, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 7 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:21.888545 30595 raft_consensus.cc:2468] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2.
I20251025 20:04:21.889426 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:21.896802 30595 raft_consensus.cc:1240] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 5. Current term is 6. Ops: []
I20251025 20:04:21.899427 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7294f698d29e4c919745b460e4e25eeb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:21.900389 30599 raft_consensus.cc:2468] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
I20251025 20:04:21.906911 30501 catalog_manager.cc:5649] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 2 to 3, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 3 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:21.909561 30501 catalog_manager.cc:5649] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 3 to 4, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:21.911548 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:21.912846 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:04:21.910455 32174 log.cc:927] Time spent T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20251025 20:04:21.831060 32183 consensus_queue.cc:1059] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
W20251025 20:04:21.892014 32149 consensus_meta.cc:287] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.622s user 0.010s sys 0.003s
I20251025 20:04:21.919052 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:21.921408 30603 raft_consensus.cc:1240] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:21.919113 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:21.866680 31801 consensus_queue.cc:1048] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
W20251025 20:04:21.913265 32175 log.cc:927] Time spent T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20251025 20:04:21.930821 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:21.930284 30501 catalog_manager.cc:5649] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 4, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:21.877842 32184 consensus_queue.cc:1059] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20251025 20:04:21.945204 30716 leader_election.cc:304] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:21.863687 32171 consensus_queue.cc:1048] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20251025 20:04:21.929076 31993 consensus_queue.cc:1059] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20251025 20:04:21.926430 32149 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.896907 32131 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.963102 30601 raft_consensus.cc:1240] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:21.963774 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:21.965175 32131 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.971441 32149 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.972329 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:21.974082 32177 raft_consensus.cc:2749] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251025 20:04:21.976696 32065 raft_consensus.cc:3055] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Stepping down as leader of term 4
W20251025 20:04:21.976720 32185 log.cc:927] Time spent T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20251025 20:04:21.979044 32065 raft_consensus.cc:740] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:21.982743 32115 consensus_meta.cc:287] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.822s user 0.018s sys 0.000s
I20251025 20:04:21.983202 32115 raft_consensus.cc:515] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:21.984302 32180 log.cc:927] Time spent T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.298s user 0.004s sys 0.000s
I20251025 20:04:21.985303 32115 leader_election.cc:290] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 7 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:21.929733 32163 leader_election.cc:290] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:21.982798 32065 consensus_queue.cc:260] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:21.995623 32025 ts_tablet_manager.cc:1916] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:04:21.995822 32065 raft_consensus.cc:3060] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:21.996492 32095 tablet_bootstrap.cc:654] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e: Neither blocks nor log segments found. Creating new log.
W20251025 20:04:21.986421 32146 consensus_meta.cc:287] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.909s user 0.010s sys 0.005s
I20251025 20:04:21.948849 32072 leader_election.cc:290] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.025374 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.026451 30601 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:22.018009 30501 catalog_manager.cc:5649] T 53eca06f855d4e699f10ab84a2ba0981 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 3, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 3 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
W20251025 20:04:22.027734 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 82635.6us
W20251025 20:04:22.016336 32157 consensus_meta.cc:287] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.666s user 0.012s sys 0.000s
I20251025 20:04:22.034368 32157 raft_consensus.cc:515] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.035383 32163 consensus_queue.cc:1059] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:04:22.036634 32157 leader_election.cc:290] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:22.036933 32187 log.cc:927] Time spent T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20251025 20:04:22.039234 32072 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:22.039773 32072 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:21.999136 31970 consensus_queue.cc:1175] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "583af532dff04e8ebba45013323d516d" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 4 index: 3 } }
I20251025 20:04:21.867923 32100 rpcz_store.cc:276] 1025 20:04:18.474554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:18.697730 (+223176us) service_pool.cc:225] Handling call
1025 20:04:19.997457 (+1299727us) ts_tablet_manager.cc:1151] Deleting tablet e8c971a6d4854180b4d243fa79fbbc11
1025 20:04:20.000563 (+ 3106us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 20:04:21.266576 (+1266013us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:21.695761 (+429185us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:21.818272 (+122511us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":494999,"tablet-delete.queue_time_us":1290026,"thread_start_us":810604,"threads_started":1}
I20251025 20:04:22.043731 32100 tablet_replica.cc:333] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: stopping tablet replica
W20251025 20:04:21.790858 32051 ts_tablet_manager.cc:1469] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957: Trace:
1025 20:04:16.741420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.744034 (+ 2614us) service_pool.cc:225] Handling call
1025 20:04:16.746325 (+ 2291us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:16.746469 (+ 144us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:17.005716 (+259247us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:17.224565 (+218849us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:17.346386 (+121821us) inbound_call.cc:173] Queueing success response
1025 20:04:20.011778 (+2665392us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1025 20:04:21.025018 (+1013240us) ts_tablet_manager.cc:1435] Starting tablet replica
1025 20:04:21.282952 (+257934us) tablet_replica.cc:261] Starting instrumentation
1025 20:04:21.455606 (+172654us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":73413,"raft.queue_time_us":1766,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"tablet-open.queue_time_us":2656961}
W20251025 20:04:22.044411 32132 consensus_meta.cc:287] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.925s user 0.009s sys 0.001s
I20251025 20:04:22.045202 32051 tablet_bootstrap.cc:492] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:04:22.044829 32132 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.045996 30602 raft_consensus.cc:1240] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 8 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 7. Current term is 8. Ops: []
I20251025 20:04:22.046226 30600 raft_consensus.cc:1240] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:22.040086 32177 raft_consensus.cc:2749] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251025 20:04:22.019704 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:22.053608 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:22.055284 30716 leader_election.cc:304] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:04:22.055505 30719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:22.056188 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:22.033811 32144 consensus_meta.cc:287] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.935s user 0.008s sys 0.004s
I20251025 20:04:22.064059 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:22.072767 30601 raft_consensus.cc:2410] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:22.063586 31965 raft_consensus.cc:493] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:22.074323 30716 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:22.044847 32100 raft_consensus.cc:2243] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:04:22.041766 32072 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.074738 31414 raft_consensus.cc:3055] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Stepping down as leader of term 5
I20251025 20:04:22.088989 31414 raft_consensus.cc:740] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:22.028333 32146 raft_consensus.cc:515] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:22.070451 30567 consensus_peers.cc:597] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 -> Peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Couldn't send request to peer 583af532dff04e8ebba45013323d516d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:04:22.085464 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.081346 30501 catalog_manager.cc:5649] T d33d554021204714a7a71332612e2070 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 3 to 4, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:22.089838 31414 consensus_queue.cc:260] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.093703 30604 raft_consensus.cc:2393] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.100368 31414 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Advancing to term 6
W20251025 20:04:22.087199 32019 consensus_meta.cc:287] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 1.023s user 0.011s sys 0.011s
I20251025 20:04:22.075489 32132 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 8 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.073832 30486 catalog_manager.cc:4985] TS 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): tablet e8c971a6d4854180b4d243fa79fbbc11 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) successfully deleted
I20251025 20:04:22.108104 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.079208 30749 raft_consensus.cc:2468] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:22.119817 30603 raft_consensus.cc:1275] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:22.124366 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:22.121734 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:22.126343 30747 raft_consensus.cc:2368] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4.
I20251025 20:04:22.127351 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.127399 30750 raft_consensus.cc:1240] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 2. Current term is 3. Ops: []
I20251025 20:04:22.128640 32072 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:22.128887 30601 raft_consensus.cc:1275] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:22.129194 32072 raft_consensus.cc:515] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:22.120050 30486 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 54051.1us
I20251025 20:04:22.130518 30596 raft_consensus.cc:1240] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 5. Current term is 6. Ops: []
I20251025 20:04:22.131278 32072 leader_election.cc:290] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.132241 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7294f698d29e4c919745b460e4e25eeb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:22.132465 32142 raft_consensus.cc:2749] T 380df7888a29498d99ec3fe64410a542 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251025 20:04:22.098455 32108 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.103890 32183 raft_consensus.cc:3060] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:22.074229 31965 raft_consensus.cc:515] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.082185 32144 raft_consensus.cc:515] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.149784 30645 leader_election.cc:304] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:22.114879 32100 raft_consensus.cc:2272] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:04:22.147284 30501 catalog_manager.cc:5649] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:22.157014 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:22.149711 32184 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:22.180382 30751 raft_consensus.cc:1240] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:22.114172 32019 raft_consensus.cc:515] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:22.183005 30501 catalog_manager.cc:5649] T 380df7888a29498d99ec3fe64410a542 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 3 to 4, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 4 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:22.184006 30719 leader_election.cc:304] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:22.190318 30501 catalog_manager.cc:5649] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 3 to 4, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:22.152733 32072 raft_consensus.cc:493] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:22.196470 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.199383 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:22.178356 32140 consensus_meta.cc:287] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.996s user 0.006s sys 0.008s
I20251025 20:04:22.199926 30600 raft_consensus.cc:2393] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 6: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.177831 32131 raft_consensus.cc:2804] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:04:22.177670 30749 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Advancing to term 7
I20251025 20:04:22.201217 30501 catalog_manager.cc:5649] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 3 to 4, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 583af532dff04e8ebba45013323d516d (127.28.192.194). New cstate: current_term: 4 leader_uuid: "583af532dff04e8ebba45013323d516d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:22.208699 30748 raft_consensus.cc:2368] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5.
I20251025 20:04:22.183686 32062 raft_consensus.cc:493] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:22.220554 32131 raft_consensus.cc:697] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:22.229985 31965 leader_election.cc:290] T 3a03f8a32da8499dbc0e2a426d214001 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.242852 31993 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:22.243407 31993 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.245352 30500 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39186 (request call id 99) took 8732 ms. Trace:
I20251025 20:04:22.245533 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8afaad4899545a9a4169026b63c0699" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:22.246869 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:22.247301 32146 leader_election.cc:290] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:22.207335 32093 consensus_meta.cc:287] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 1.157s user 0.012s sys 0.009s
I20251025 20:04:22.248857 32019 leader_election.cc:290] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.249142 32093 raft_consensus.cc:515] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:22.250128 32121 consensus_meta.cc:287] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 1.050s user 0.011s sys 0.004s
I20251025 20:04:22.250593 32121 raft_consensus.cc:515] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.251328 32093 leader_election.cc:290] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.252568 32121 leader_election.cc:290] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.253253 30597 raft_consensus.cc:1240] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 1. Current term is 2. Ops: []
I20251025 20:04:22.245915 30500 rpcz_store.cc:276] 1025 20:04:13.476514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:13.476895 (+ 381us) service_pool.cc:225] Handling call
1025 20:04:22.204979 (+8728084us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 20:04:17.873682 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 20:04:17.921965 (+ 48283us) write_op.cc:432] Acquiring schema lock in shared mode
1025 20:04:17.922001 (+ 36us) write_op.cc:435] Acquired schema lock
1025 20:04:17.922019 (+ 18us) tablet.cc:660] Decoding operations
1025 20:04:17.926694 (+ 4675us) write_op.cc:620] Acquiring the partition lock for write op
1025 20:04:17.926768 (+ 74us) write_op.cc:641] Partition lock acquired for write op
1025 20:04:17.926812 (+ 44us) tablet.cc:683] Acquiring locks for 18 operations
1025 20:04:17.928199 (+ 1387us) tablet.cc:699] Row locks acquired
1025 20:04:17.928215 (+ 16us) write_op.cc:260] PREPARE: finished
1025 20:04:17.928451 (+ 236us) write_op.cc:270] Start()
1025 20:04:17.928580 (+ 129us) write_op.cc:276] Timestamp: P: 1761422657928397 usec, L: 0
1025 20:04:17.928598 (+ 18us) op_driver.cc:348] REPLICATION: starting
1025 20:04:17.929442 (+ 844us) log.cc:844] Serialized 7191 byte log entry
1025 20:04:20.326791 (+2397349us) op_driver.cc:464] REPLICATION: finished
1025 20:04:20.448311 (+121520us) write_op.cc:301] APPLY: starting
1025 20:04:20.448459 (+ 148us) tablet.cc:1365] starting BulkCheckPresence
1025 20:04:20.448978 (+ 519us) tablet.cc:1368] finished BulkCheckPresence
1025 20:04:20.449134 (+ 156us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 20:04:20.451481 (+ 2347us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 20:04:20.451869 (+ 388us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
1025 20:04:20.451937 (+ 68us) write_op.cc:312] APPLY: finished
1025 20:04:20.452532 (+ 595us) log.cc:844] Serialized 124 byte log entry
1025 20:04:20.452746 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:04:20.473822 (+ 21076us) write_op.cc:454] Released schema lock
1025 20:04:20.482038 (+ 8216us) write_op.cc:341] FINISH: Updating metrics
1025 20:04:20.583235 (+101197us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:04:20.585058 (+ 1823us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1200,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":98030,"apply.run_cpu_time_us":5559,"apply.run_wall_time_us":61155,"num_ops":18,"prepare.queue_time_us":350392,"prepare.run_cpu_time_us":44449,"prepare.run_wall_time_us":1794440,"raft.queue_time_us":436944,"raft.run_cpu_time_us":14593,"raft.run_wall_time_us":148572,"replication_time_us":2351551,"thread_start_us":2281845,"threads_started":4,"wal-append.queue_time_us":1837177,"wal-append.run_cpu_time_us":38099,"wal-append.run_wall_time_us":1689311}]]}
I20251025 20:04:22.254534 30597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34874 (request call id 889) took 1099 ms. Trace:
W20251025 20:04:22.219615 32192 log.cc:927] Time spent T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20251025 20:04:22.196199 32163 consensus_queue.cc:1059] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
W20251025 20:04:22.231079 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:04:22.225675 30501 catalog_manager.cc:5649] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 2, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:22.258905 30501 catalog_manager.cc:5649] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d reported cstate change: term changed from 1 to 2, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 2 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:22.262305 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.263182 30598 raft_consensus.cc:2393] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 5: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:22.202661 32072 raft_consensus.cc:515] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.202262 30738 tablet_service.cc:1552] Processing DeleteTablet for tablet e8c971a6d4854180b4d243fa79fbbc11 with delete_type TABLET_DATA_DELETED (Replaced by f69f57fe417f492d9b82ed651b9d70c1 at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:53746
I20251025 20:04:22.233203 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.264030 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.265014 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.265813 30600 raft_consensus.cc:2393] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 5: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.266283 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.266659 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.267074 30596 raft_consensus.cc:2393] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 5: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.267803 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.268604 30596 raft_consensus.cc:2393] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.269331 32100 ts_tablet_manager.cc:1916] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:04:22.269722 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.270834 32144 leader_election.cc:290] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.271283 32072 leader_election.cc:290] T 6973d3401da04486853abdd647a5795a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.241976 30744 raft_consensus.cc:1240] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251025 20:04:22.272116 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:22.241268 30748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 568) took 1196 ms (client timeout 1508 ms). Trace:
I20251025 20:04:22.272908 30744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 576) took 1179 ms. Trace:
W20251025 20:04:22.273206 30748 rpcz_store.cc:269] 1025 20:04:21.044470 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:21.044808 (+ 338us) service_pool.cc:225] Handling call
1025 20:04:21.090487 (+ 45679us) spinlock_profiling.cc:243] Waited 6 ms on lock 0x7b58000f2198. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd81e8ce381 00007fd81ea50451 00007fd81ea50302 00007fd8278e9ad9 00007fd826a6cd30 00007fd826b3584b 00007fd826b358d1 00007fd826ae3660 00007fd826ae2f44 00007fd820080834 00007fd8200807c2
1025 20:04:22.241028 (+1150541us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400209c54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:22.241180 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2502401664}
I20251025 20:04:22.273360 30744 rpcz_store.cc:276] 1025 20:04:21.093202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:21.108207 (+ 15005us) service_pool.cc:225] Handling call
1025 20:04:22.241641 (+1133434us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:22.271919 (+ 30278us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:22.272671 (+ 752us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400209c50. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3944d 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:22.272812 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2507113344}
I20251025 20:04:22.274709 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:22.274838 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:22.275844 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:22.276432 30748 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 5.
I20251025 20:04:22.277230 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:22.279392 32177 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:22.279970 32177 raft_consensus.cc:515] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.180526 32184 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.288715 32184 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.183267 32098 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.265844 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.291491 32098 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.298290 30750 raft_consensus.cc:2368] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5.
I20251025 20:04:22.238977 32062 raft_consensus.cc:515] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.300148 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:22.300887 32062 leader_election.cc:290] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:22.302521 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 118611us
I20251025 20:04:22.272908 30599 raft_consensus.cc:2393] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:22.221055 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:22.270860 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.255641 32194 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:22.242425 32051 tablet_bootstrap.cc:654] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:22.265069 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.267747 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.240139 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:22.186180 32179 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.271224 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.291536 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.254878 30597 rpcz_store.cc:276] 1025 20:04:21.155085 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:21.158078 (+ 2993us) service_pool.cc:225] Handling call
1025 20:04:21.172555 (+ 14477us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:22.253636 (+1081081us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:22.254281 (+ 645us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540038e354. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:22.254474 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2484870016}
I20251025 20:04:22.209133 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.269357 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.305076 30602 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:22.205233 32140 raft_consensus.cc:515] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.203575 32096 raft_consensus.cc:3055] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:22.314831 30753 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:22.350612 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.234647 32152 raft_consensus.cc:493] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:22.371547 30604 raft_consensus.cc:2393] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.245610 31993 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.406364 30673 raft_consensus.cc:1275] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251025 20:04:22.508028 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 155864us
I20251025 20:04:22.509406 30603 raft_consensus.cc:2393] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 5: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:22.334914 32015 raft_consensus.cc:2804] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:04:22.512508 30596 raft_consensus.cc:2393] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.469688 30600 raft_consensus.cc:1275] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:22.482815 30676 raft_consensus.cc:1240] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:22.437268 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:22.420704 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:22.482234 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.526988 30599 raft_consensus.cc:2393] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 6: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
W20251025 20:04:22.528563 32099 consensus_meta.cc:287] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.736s user 0.012s sys 0.003s
I20251025 20:04:22.537528 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:22.491048 32168 consensus_meta.cc:287] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.748s user 0.014s sys 0.000s
W20251025 20:04:22.540220 30645 leader_election.cc:336] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Timed out: RequestConsensusVote RPC to 127.28.192.195:46541 timed out after 1.508s (SENT)
I20251025 20:04:22.542116 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:22.547832 30673 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:22.543243 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:22.420322 32173 raft_consensus.cc:493] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:22.498816 32177 leader_election.cc:290] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.528782 32015 raft_consensus.cc:493] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:22.515319 32096 raft_consensus.cc:740] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:22.546129 30746 raft_consensus.cc:2468] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 3.
I20251025 20:04:22.556016 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:22.559396 30752 raft_consensus.cc:1240] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251025 20:04:22.561290 32140 leader_election.cc:290] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.571674 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:22.572659 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.576532 30670 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 5
I20251025 20:04:22.536451 32062 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:22.333426 32194 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.571681 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:22.578650 30674 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:22.582302 30674 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
W20251025 20:04:22.578927 30716 leader_election.cc:343] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:04:22.508186 32163 consensus_queue.cc:1059] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20251025 20:04:22.583115 30673 raft_consensus.cc:2468] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:22.583603 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.584251 30716 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:22.585467 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78afc6e697b1482391aa5245bee3d881" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.586123 30602 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:22.525974 31970 consensus_queue.cc:1048] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20251025 20:04:22.539482 32072 raft_consensus.cc:493] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:22.590622 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.590719 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:22.592996 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 52826.7us
I20251025 20:04:22.524715 32152 raft_consensus.cc:515] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:22.596678 30602 raft_consensus.cc:2468] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:22.597210 30675 raft_consensus.cc:1275] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:22.536127 32098 consensus_queue.cc:1048] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
W20251025 20:04:22.603556 30645 leader_election.cc:343] T d8afaad4899545a9a4169026b63c0699 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:04:22.576471 32015 raft_consensus.cc:3060] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:22.601733 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.605983 30598 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:22.606073 30645 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:22.598501 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.604784 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.607025 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.607064 30597 raft_consensus.cc:3060] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:22.553411 31993 consensus_queue.cc:1059] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20251025 20:04:22.601787 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13be59a31cda407db6fa50a400f37e1f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.615309 30604 raft_consensus.cc:3060] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 4
I20251025 20:04:22.616212 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.617039 30600 raft_consensus.cc:2393] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.616916 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.618656 30595 raft_consensus.cc:2393] T 97f7928c55054a2f809df6e4ee99800a P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.621243 30599 raft_consensus.cc:1275] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:22.572487 32173 raft_consensus.cc:515] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.595714 32199 consensus_queue.cc:1059] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20251025 20:04:22.594569 32062 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.614866 32065 raft_consensus.cc:2749] T 6cbb781b2e4441b4be9b53f7c0947525 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251025 20:04:22.598438 32072 raft_consensus.cc:515] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.658149 30748 raft_consensus.cc:1275] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:22.614354 32115 consensus_queue.cc:1059] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20251025 20:04:22.668908 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.613703 32111 consensus_queue.cc:1059] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20251025 20:04:22.671655 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.580009 32131 consensus_queue.cc:237] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.618108 32093 consensus_queue.cc:1059] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20251025 20:04:22.623021 32096 consensus_queue.cc:260] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.682058 30599 raft_consensus.cc:2393] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 6: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.682135 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.679967 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.687709 30601 raft_consensus.cc:2393] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 6: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.677233 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.674863 30600 raft_consensus.cc:2376] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 583af532dff04e8ebba45013323d516d in term 2. Re-sending same reply.
I20251025 20:04:22.691109 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "827742f9e2064e96a4fdcbd6a9b9a4a2" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.695789 30738 tablet_service.cc:1552] Processing DeleteTablet for tablet fea0c74c9c0c4a69afe072c45c4cbe99 with delete_type TABLET_DATA_DELETED (Replaced by 7294f698d29e4c919745b460e4e25eeb at 2025-10-25 20:04:12 UTC) from {username='slave'} at 127.0.0.1:53746
I20251025 20:04:22.662941 32158 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
W20251025 20:04:22.703138 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 97131.9us
I20251025 20:04:22.707255 30645 leader_election.cc:400] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5.
I20251025 20:04:22.707811 30645 leader_election.cc:403] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20251025 20:04:22.691962 32082 consensus_meta.cc:287] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.846s user 0.012s sys 0.002s
W20251025 20:04:22.700039 32196 log.cc:927] Time spent T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.217s user 0.001s sys 0.000s
I20251025 20:04:22.718755 30719 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
W20251025 20:04:22.704128 32200 log.cc:927] Time spent T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
I20251025 20:04:22.757323 30599 raft_consensus.cc:1275] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251025 20:04:22.759485 30749 consensus_meta.cc:287] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.523s user 0.010s sys 0.000s
W20251025 20:04:22.757388 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 134647us
I20251025 20:04:22.759886 30749 raft_consensus.cc:2468] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 7.
I20251025 20:04:22.760663 30751 raft_consensus.cc:1240] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 7 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 6. Current term is 7. Ops: []
I20251025 20:04:22.762676 30745 raft_consensus.cc:2468] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:22.710254 32179 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.770048 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:22.770346 30716 leader_election.cc:304] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:04:22.772682 32198 log.cc:927] Time spent T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.242s user 0.001s sys 0.000s
I20251025 20:04:22.772589 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.780069 30599 raft_consensus.cc:2393] T db5a3b726f79432baf8b25596709ca04 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 2: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.747546 32111 raft_consensus.cc:2804] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:22.778549 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20fe84e124304a7a9d92910cfa619beb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:22.823339 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 116329us
I20251025 20:04:22.824313 30747 raft_consensus.cc:2468] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
W20251025 20:04:22.824410 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 44209.5us
I20251025 20:04:22.765380 32163 raft_consensus.cc:2804] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:22.825881 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.827597 30716 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957
I20251025 20:04:22.829684 30716 leader_election.cc:304] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:22.829771 30645 leader_election.cc:304] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:22.713333 32158 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.832792 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:22.764969 31993 consensus_queue.cc:1048] T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.064s
W20251025 20:04:22.825173 32196 log.cc:927] Time spent T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20251025 20:04:22.786814 32111 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:22.823333 31965 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:22.839581 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da8a0eaa68044835a4cf94340ca25ac5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.850970 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d23f4ba1a4b411baf57f59f30489265" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.854100 30599 raft_consensus.cc:2393] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 5: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:22.828560 32212 raft_consensus.cc:2749] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:22.862877 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8afaad4899545a9a4169026b63c0699" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.835350 32098 raft_consensus.cc:2804] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:22.865796 30749 raft_consensus.cc:1275] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:22.870303 30749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 602) took 1061 ms. Trace:
I20251025 20:04:22.872468 30604 raft_consensus.cc:2468] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:22.870703 30749 rpcz_store.cc:276] 1025 20:04:21.808842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:22.837309 (+1028467us) service_pool.cc:225] Handling call
1025 20:04:22.865301 (+ 27992us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:22.869996 (+ 4695us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:22.870240 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:22.836169 32152 leader_election.cc:290] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.878814 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d399a886ee4166a2ea1b4410f9425a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.849560 32215 raft_consensus.cc:3055] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:22.856559 32111 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:22.853225 32163 raft_consensus.cc:697] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:22.882207 30670 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:22.837535 32194 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.915666 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:22.916417 30749 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:22.883417 30598 raft_consensus.cc:2468] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:22.859251 31965 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.887044 32158 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.881587 32168 raft_consensus.cc:2804] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election won for term 4
W20251025 20:04:22.891265 32208 log.cc:927] Time spent T 00000000000000000000000000000000 P 5b68518dd72d4e87a5c2dcf7c969fe49: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20251025 20:04:22.892989 30597 raft_consensus.cc:2468] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:22.893746 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813ab494bf654127a41723bb256c1ede" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.882376 32173 leader_election.cc:290] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:22.899075 32216 raft_consensus.cc:493] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:22.904754 30752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 577) took 1804 ms. Trace:
I20251025 20:04:22.858924 32179 raft_consensus.cc:2804] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Leader election won for term 7
I20251025 20:04:22.868551 32098 raft_consensus.cc:493] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:22.937876 32098 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:22.938841 32140 raft_consensus.cc:3055] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:22.939421 32140 raft_consensus.cc:740] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:22.940560 32140 consensus_queue.cc:260] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.943053 30677 raft_consensus.cc:1275] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251025 20:04:22.945297 32214 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:22.946019 32214 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.948101 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f69f57fe417f492d9b82ed651b9d70c1" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:22.948967 32214 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.910177 30746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 587) took 1181 ms. Trace:
I20251025 20:04:22.911422 30753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 605) took 1037 ms. Trace:
I20251025 20:04:22.912096 32095 tablet_bootstrap.cc:492] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e: No bootstrap required, opened a new log
W20251025 20:04:22.913177 32183 consensus_meta.cc:287] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.763s user 0.012s sys 0.000s
W20251025 20:04:22.918805 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 89202.2us
I20251025 20:04:22.954181 30749 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 5.
I20251025 20:04:22.954363 30716 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:22.952066 30753 rpcz_store.cc:276] 1025 20:04:21.874191 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:21.874612 (+ 421us) service_pool.cc:225] Handling call
1025 20:04:22.911189 (+1036577us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b54002d8054. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:22.911341 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":738566016}
I20251025 20:04:22.951257 30746 rpcz_store.cc:276] 1025 20:04:21.729016 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:21.729232 (+ 216us) service_pool.cc:225] Handling call
1025 20:04:22.909925 (+1180693us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b54001e7154. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:22.910088 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1852486016}
I20251025 20:04:22.920007 30600 raft_consensus.cc:1240] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:22.921497 30748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 598) took 1116 ms. Trace:
I20251025 20:04:22.927990 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:22.928639 30604 raft_consensus.cc:2368] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5.
I20251025 20:04:22.898372 32072 leader_election.cc:290] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.881918 32215 raft_consensus.cc:740] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:22.898437 32062 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.870679 32065 consensus_queue.cc:1048] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.051s
I20251025 20:04:22.914626 32115 raft_consensus.cc:2749] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5.
I20251025 20:04:22.934886 32168 raft_consensus.cc:697] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:22.937218 31965 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:22.951848 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:22.933188 30752 rpcz_store.cc:276] 1025 20:04:21.100381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:21.123912 (+ 23531us) service_pool.cc:225] Handling call
1025 20:04:21.154301 (+ 30389us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:22.573728 (+1419427us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:22.904358 (+330630us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54001437d4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:22.904655 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3230637312}
I20251025 20:04:22.956604 30748 rpcz_store.cc:276] 1025 20:04:21.800860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:22.333440 (+532580us) service_pool.cc:225] Handling call
1025 20:04:22.466843 (+133403us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:22.658768 (+191925us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:22.916592 (+257824us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54002a0b54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:22.916855 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":438624128}
I20251025 20:04:22.957644 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:22.964570 32214 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:22.975409 30675 raft_consensus.cc:2468] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 5.
I20251025 20:04:22.965786 32095 ts_tablet_manager.cc:1403] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e: Time spent bootstrapping tablet: real 1.246s user 0.027s sys 0.015s
I20251025 20:04:22.959412 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:22.960490 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:22.961458 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 102952us
I20251025 20:04:22.961690 30645 leader_election.cc:304] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:22.959131 30716 leader_election.cc:304] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
@ 0x7fd81ec7fcf0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
I20251025 20:04:22.953876 32216 raft_consensus.cc:515] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:22.977313 30746 raft_consensus.cc:1275] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251025 20:04:22.977353 30567 leader_election.cc:343] T f69f57fe417f492d9b82ed651b9d70c1 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
W20251025 20:04:23.093501 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 116200us
I20251025 20:04:23.093534 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d77f4dd14642b7bdb7299e84673c03" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:23.094364 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 132791us
I20251025 20:04:23.093680 30746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 581) took 1275 ms. Trace:
I20251025 20:04:23.096714 30746 rpcz_store.cc:276] 1025 20:04:21.818550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:22.964051 (+1145501us) service_pool.cc:225] Handling call
1025 20:04:22.976963 (+ 12912us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:23.093357 (+116394us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:23.093615 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:23.016186 32140 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:23.098011 32213 consensus_queue.cc:1059] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:23.098836 30753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 606) took 1182 ms. Trace:
I20251025 20:04:23.098332 32121 raft_consensus.cc:3055] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 7 LEADER]: Stepping down as leader of term 7
I20251025 20:04:23.094620 30601 raft_consensus.cc:2368] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:23.099613 32121 raft_consensus.cc:740] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:23.099229 30753 rpcz_store.cc:276] 1025 20:04:21.916234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:23.097852 (+1181618us) service_pool.cc:225] Handling call
1025 20:04:23.098240 (+ 388us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:23.098468 (+ 228us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:23.098494 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:23.098542 (+ 48us) raft_consensus.cc:1645] Marking committed up to 3
1025 20:04:23.098575 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:23.098634 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:23.098750 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:23.100960 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.100736 32121 consensus_queue.cc:260] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.101730 30601 raft_consensus.cc:2393] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 5: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:23.047135 32206 raft_consensus.cc:3055] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Stepping down as leader of term 1
I20251025 20:04:23.102299 30749 raft_consensus.cc:2468] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
I20251025 20:04:23.102437 32206 raft_consensus.cc:740] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:23.101857 32152 raft_consensus.cc:493] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:23.103194 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.103772 30749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 608) took 1076 ms. Trace:
I20251025 20:04:23.103323 32206 consensus_queue.cc:260] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.103713 32152 raft_consensus.cc:515] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.104550 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.049032 32108 raft_consensus.cc:3055] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:23.104984 32206 raft_consensus.cc:3060] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:04:23.105450 32108 raft_consensus.cc:740] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:23.099068 32213 raft_consensus.cc:3055] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:23.105618 30601 raft_consensus.cc:2468] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 5.
I20251025 20:04:23.106109 32213 raft_consensus.cc:740] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:23.106455 32108 consensus_queue.cc:260] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.107002 32213 consensus_queue.cc:260] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.107579 32108 raft_consensus.cc:3060] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:23.108372 32213 raft_consensus.cc:3060] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:23.108626 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.109998 32152 leader_election.cc:290] T bc001e0fba604d68bfa98fc25d70bbbb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:23.048238 32111 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.024662 32163 consensus_queue.cc:237] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:23.110769 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0a9871641274e83b2300f3ad8095add" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.027601 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec9d5d5c538f4253bcec6b8b23843ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.119839 30604 raft_consensus.cc:2393] T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
I20251025 20:04:23.018743 30745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 578) took 1469 ms. Trace:
I20251025 20:04:23.111672 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:22.962886 32173 raft_consensus.cc:2749] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251025 20:04:22.962707 32183 raft_consensus.cc:2749] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:23.112190 30751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 599) took 1311 ms. Trace:
I20251025 20:04:22.984849 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fea0c74c9c0c4a69afe072c45c4cbe99" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.105567 30749 rpcz_store.cc:276] 1025 20:04:22.027639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:23.102972 (+1075333us) service_pool.cc:225] Handling call
1025 20:04:23.103277 (+ 305us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:23.103465 (+ 188us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:23.103489 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:23.103532 (+ 43us) raft_consensus.cc:1645] Marking committed up to 1
1025 20:04:23.103562 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:23.103618 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:23.103718 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:23.114183 30747 raft_consensus.cc:1240] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:23.005738 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:23.120649 30745 rpcz_store.cc:276] 1025 20:04:21.549134 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:21.549293 (+ 159us) service_pool.cc:225] Handling call
1025 20:04:23.018473 (+1469180us) spinlock_profiling.cc:243] Waited 974 ms on lock 0x7b540021fa54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:23.018653 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2239968640}
I20251025 20:04:23.133563 30747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 609) took 1081 ms. Trace:
I20251025 20:04:23.134080 30747 rpcz_store.cc:276] 1025 20:04:22.052371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:23.113504 (+1061133us) service_pool.cc:225] Handling call
1025 20:04:23.113881 (+ 377us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:23.133281 (+ 19400us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:23.133503 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:23.132416 30567 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:23.136806 30645 leader_election.cc:304] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:23.132390 30751 rpcz_store.cc:276] 1025 20:04:21.801179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:22.543638 (+742459us) service_pool.cc:225] Handling call
1025 20:04:22.760486 (+216848us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:22.761010 (+ 524us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:23.111974 (+350964us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b5400239e50. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3944d 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:23.112128 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":427440384}
I20251025 20:04:23.134991 30745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 613) took 1021 ms. Trace:
I20251025 20:04:23.138336 30753 raft_consensus.cc:1240] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:23.139482 30745 rpcz_store.cc:276] 1025 20:04:22.113727 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:23.133649 (+1019922us) service_pool.cc:225] Handling call
1025 20:04:23.134364 (+ 715us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:23.134633 (+ 269us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:23.134663 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1025 20:04:23.134705 (+ 42us) raft_consensus.cc:1645] Marking committed up to 2
1025 20:04:23.134746 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:23.134805 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:23.134935 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 20:04:23.092816 32072 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:23.011113 32210 raft_consensus.cc:3055] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:23.004833 32214 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.093693 32179 raft_consensus.cc:697] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 7 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:23.120517 32131 raft_consensus.cc:2804] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:22.968964 32070 consensus_queue.cc:1048] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.169s
I20251025 20:04:22.967792 32166 consensus_queue.cc:1048] T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.173s
W20251025 20:04:23.146973 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:23.045557 30744 raft_consensus.cc:2468] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 6.
W20251025 20:04:23.149071 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 190019us
I20251025 20:04:23.136544 32100 ts_tablet_manager.cc:1929] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251025 20:04:23.150158 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.150357 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:23.122334 32152 raft_consensus.cc:493] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:23.153239 30598 raft_consensus.cc:2393] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:23.146591 32144 raft_consensus.cc:3055] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Stepping down as leader of term 5
I20251025 20:04:23.156765 30672 raft_consensus.cc:1275] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:23.157776 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43d8dfe478a4430da93c214b20963f9f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:23.016598 32098 raft_consensus.cc:515] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.161343 32179 consensus_queue.cc:237] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.162036 32111 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:23.162921 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:23.157228 32100 log.cc:1199] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: Deleting WAL directory at /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-1-root/wals/fea0c74c9c0c4a69afe072c45c4cbe99
I20251025 20:04:23.156823 32194 raft_consensus.cc:2804] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:23.163705 32214 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:23.165341 32216 leader_election.cc:290] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:23.202133 32216 raft_consensus.cc:2804] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:23.202205 32194 raft_consensus.cc:697] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:23.202615 32216 raft_consensus.cc:493] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:23.202903 32025 ts_tablet_manager.cc:1929] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.2
I20251025 20:04:23.203044 32216 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:23.203398 32025 log.cc:1199] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: Deleting WAL directory at /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-2-root/wals/da8a0eaa68044835a4cf94340ca25ac5
I20251025 20:04:23.204288 32025 ts_tablet_manager.cc:1950] T da8a0eaa68044835a4cf94340ca25ac5 P 783f8fd6d80141b7912f66a954a6f92e: Deleting consensus metadata
W20251025 20:04:23.204429 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 51122.9us
I20251025 20:04:23.205292 32025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53746 (request call id 61) took 6599 ms. Trace:
I20251025 20:04:23.006181 32215 consensus_queue.cc:260] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } }
I20251025 20:04:23.166230 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:23.167657 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:23.157881 32144 raft_consensus.cc:740] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:23.048704 32168 consensus_queue.cc:237] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.144874 32225 consensus_queue.cc:1048] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
I20251025 20:04:23.174016 31965 raft_consensus.cc:2804] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:23.174536 30677 raft_consensus.cc:2468] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:23.177682 32051 tablet_bootstrap.cc:492] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957: No bootstrap required, opened a new log
I20251025 20:04:23.149358 32072 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.180311 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:23.185727 30750 raft_consensus.cc:2468] T 6973d3401da04486853abdd647a5795a P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:23.009656 32015 raft_consensus.cc:515] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.143440 32210 raft_consensus.cc:740] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:23.220166 30716 leader_election.cc:304] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:23.205622 32025 rpcz_store.cc:276] 1025 20:04:16.606069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.612543 (+ 6474us) service_pool.cc:225] Handling call
1025 20:04:21.232100 (+4619557us) ts_tablet_manager.cc:1151] Deleting tablet da8a0eaa68044835a4cf94340ca25ac5
1025 20:04:21.232209 (+ 109us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 20:04:22.763317 (+1531108us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:23.202583 (+439266us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:23.205228 (+ 2645us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":639596,"tablet-delete.queue_time_us":4606323}
I20251025 20:04:23.236716 32144 consensus_queue.cc:260] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.237298 32142 raft_consensus.cc:2749] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:23.237766 32025 tablet_replica.cc:333] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: stopping tablet replica
I20251025 20:04:23.238139 32144 raft_consensus.cc:3060] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:23.237973 32142 consensus_queue.cc:1048] T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:04:23.234412 30670 raft_consensus.cc:2410] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:23.238451 30604 raft_consensus.cc:2376] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4. Re-sending same reply.
I20251025 20:04:23.240001 32051 ts_tablet_manager.cc:1403] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957: Time spent bootstrapping tablet: real 1.195s user 0.045s sys 0.017s
I20251025 20:04:23.240077 30750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 611) took 1145 ms. Trace:
I20251025 20:04:23.224336 30752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 603) took 1390 ms. Trace:
I20251025 20:04:23.240505 30750 rpcz_store.cc:276] 1025 20:04:22.094749 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:22.299782 (+205033us) service_pool.cc:225] Handling call
1025 20:04:23.239922 (+940140us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b5400288554. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:23.240021 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1939496832}
I20251025 20:04:23.242168 30678 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
I20251025 20:04:23.217406 32098 leader_election.cc:290] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:23.241187 30752 rpcz_store.cc:276] 1025 20:04:21.834346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:22.976751 (+1142405us) service_pool.cc:225] Handling call
1025 20:04:22.977511 (+ 760us) raft_consensus.cc:1471] Updating replica for 1 ops
1025 20:04:22.977880 (+ 369us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:22.977950 (+ 70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1025 20:04:23.026220 (+ 48270us) log.cc:844] Serialized 26 byte log entry
1025 20:04:23.104448 (+ 78228us) raft_consensus.cc:1645] Marking committed up to 5
1025 20:04:23.142923 (+ 38475us) log.cc:844] Serialized 14 byte log entry
1025 20:04:23.224006 (+ 81083us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:23.224090 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:23.224137 (+ 47us) raft_consensus.cc:1685] finished
1025 20:04:23.224151 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:23.224273 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80756,"raft.queue_time_us":81482,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":37638,"thread_start_us":115847,"threads_started":2,"wal-append.queue_time_us":69708}
I20251025 20:04:23.223487 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:23.159945 32152 raft_consensus.cc:515] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.243429 30673 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:23.244859 32152 leader_election.cc:290] T 1df036864f934f2d81add520300e9ed6 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:23.231917 32023 ts_tablet_manager.cc:1929] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.2
I20251025 20:04:23.245672 32023 log.cc:1199] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: Deleting WAL directory at /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-0-root/wals/da8a0eaa68044835a4cf94340ca25ac5
I20251025 20:04:23.246548 32023 ts_tablet_manager.cc:1950] T da8a0eaa68044835a4cf94340ca25ac5 P 349d66b543bf47249ac78154b908b957: Deleting consensus metadata
I20251025 20:04:23.227245 30753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 607) took 1253 ms. Trace:
I20251025 20:04:23.224920 30746 raft_consensus.cc:1240] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:23.248920 30753 rpcz_store.cc:276] 1025 20:04:21.973871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:23.100628 (+1126757us) service_pool.cc:225] Handling call
1025 20:04:23.100952 (+ 324us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:23.140007 (+ 39055us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:23.226997 (+ 86990us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b54000bfed4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:23.227183 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85492608}
W20251025 20:04:23.208016 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:23.198218 32131 raft_consensus.cc:697] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 5 LEADER]: Becoming Leader. State: Replica: 583af532dff04e8ebba45013323d516d, State: Running, Role: LEADER
I20251025 20:04:23.205770 30719 leader_election.cc:304] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:23.267870 30746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 604) took 1414 ms. Trace:
I20251025 20:04:23.268301 30746 rpcz_store.cc:276] 1025 20:04:21.853700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:23.097606 (+1243906us) service_pool.cc:225] Handling call
1025 20:04:23.098012 (+ 406us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:23.249485 (+151473us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:23.267606 (+ 18121us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540036f954. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:23.267801 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291314944}
W20251025 20:04:23.270841 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 65157.9us
I20251025 20:04:23.207494 32100 ts_tablet_manager.cc:1950] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: Deleting consensus metadata
I20251025 20:04:23.213048 32124 tablet_bootstrap.cc:492] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:04:23.276219 30673 raft_consensus.cc:2468] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:23.279183 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7904fa0f61dd4c9fadc39faddc8f908c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:23.284324 30717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:23.227419 30744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53762 (request call id 616) took 1017 ms. Trace:
W20251025 20:04:23.198531 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 61766.4us
I20251025 20:04:23.285638 31993 raft_consensus.cc:2804] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:23.286238 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.299118 30603 raft_consensus.cc:1275] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:23.229247 32096 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:23.292047 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.302809 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:23.302847 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.298105 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:23.305437 30750 raft_consensus.cc:1275] T 393c7e0e9757478198789a4dd6c9a8a8 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:04:23.292562 30679 raft_consensus.cc:1275] T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251025 20:04:23.305833 30645 leader_election.cc:336] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.554s (SENT)
W20251025 20:04:23.292241 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:23.306340 30645 leader_election.cc:304] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.295503 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:23.248679 31965 raft_consensus.cc:493] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:23.287495 32025 raft_consensus.cc:2243] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 2 LEADER]: Raft consensus shutting down.
I20251025 20:04:23.248457 32023 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34862 (request call id 61) took 6722 ms. Trace:
I20251025 20:04:23.214257 32121 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 7 FOLLOWER]: Advancing to term 8
I20251025 20:04:23.308537 30597 raft_consensus.cc:1240] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 3. Current term is 4. Ops: []
W20251025 20:04:23.310312 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 45896.8us
I20251025 20:04:23.321238 30745 raft_consensus.cc:1240] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251025 20:04:23.322024 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
W20251025 20:04:23.324082 30645 leader_election.cc:336] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.590s (SENT)
I20251025 20:04:23.324499 30645 leader_election.cc:304] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.292172 32015 leader_election.cc:290] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:23.326507 30716 leader_election.cc:304] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:23.309485 32124 ts_tablet_manager.cc:1403] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 1.861s user 0.021s sys 0.056s
I20251025 20:04:23.329447 30599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d97d4d3534235ad57ba5d3d50de74" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:23.332130 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:04:23.336161 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.308251 31965 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:23.203403 32194 consensus_queue.cc:237] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.342991 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.345634 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7294f698d29e4c919745b460e4e25eeb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:23.346614 30602 raft_consensus.cc:3060] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:04:23.348351 30716 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:23.349450 30716 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:23.324216 32115 raft_consensus.cc:2749] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:04:23.330618 32230 raft_consensus.cc:2749] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:23.325817 32183 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:23.325970 30489 catalog_manager.cc:4985] TS 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541): tablet da8a0eaa68044835a4cf94340ca25ac5 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) successfully deleted
I20251025 20:04:23.354898 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e47b11c8d47461cac220a7cc3625f58" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.328794 31993 raft_consensus.cc:697] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:23.355036 30567 leader_election.cc:304] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d; no voters:
I20251025 20:04:23.367463 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6973d3401da04486853abdd647a5795a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.368322 30595 raft_consensus.cc:2368] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4.
I20251025 20:04:23.292981 30744 rpcz_store.cc:276] 1025 20:04:22.210246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:22.274595 (+ 64349us) service_pool.cc:225] Handling call
1025 20:04:23.227241 (+952646us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b540020ab54. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3de8a 00007fd826ae30d7 00007fd820080834 00007fd8200807c2 00007fd8200806f1 00007fd82008066c 00007fd82007f922 00007fd81fe79ecc 00007fd81fe79306 00007fd81fe78caf
1025 20:04:23.227350 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1118059904}
I20251025 20:04:23.420589 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4614bdb7f750431fa840cabccf782b4b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:23.421422 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 71376.8us
I20251025 20:04:23.422971 30719 leader_election.cc:304] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
W20251025 20:04:23.423491 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 51214.4us
I20251025 20:04:23.424419 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2eddbfc136b841098915d8ea2d82956a" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.426864 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7294f698d29e4c919745b460e4e25eeb" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:23.428804 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:23.447742 30603 raft_consensus.cc:2410] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:23.427670 30602 raft_consensus.cc:2468] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 1.
I20251025 20:04:23.357395 32142 raft_consensus.cc:2749] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:23.361790 32189 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:23.361331 32082 raft_consensus.cc:2804] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251025 20:04:23.454457 30645 leader_election.cc:336] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.606s (SENT)
I20251025 20:04:23.456373 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.454942 30645 leader_election.cc:304] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.458127 30602 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 349d66b543bf47249ac78154b908b957 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 8: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:23.460333 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.420589 32093 raft_consensus.cc:2804] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:23.421797 32100 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41086 (request call id 60) took 4571 ms. Trace:
I20251025 20:04:23.464232 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:23.464735 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f27e13dcf894a1487ed057731df6174" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.425451 32099 raft_consensus.cc:2804] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader election won for term 5
W20251025 20:04:23.474651 30645 leader_election.cc:336] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.606s (SENT)
I20251025 20:04:23.475427 30645 leader_election.cc:304] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.370208 32225 consensus_queue.cc:1048] T 393c7e0e9757478198789a4dd6c9a8a8 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
I20251025 20:04:23.453938 32025 raft_consensus.cc:2272] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 20:04:23.483520 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.457608 32082 raft_consensus.cc:493] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:23.486356 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:23.429353 30489 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 116901us
I20251025 20:04:23.466279 32093 raft_consensus.cc:493] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:23.487335 30602 raft_consensus.cc:2393] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 3: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:23.460790 32115 raft_consensus.cc:2749] T 20fe84e124304a7a9d92910cfa619beb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:23.492915 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4af0795c8ff48d9b639058ea7111f12" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.312112 32023 rpcz_store.cc:276] 1025 20:04:16.525717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.526026 (+ 309us) service_pool.cc:225] Handling call
1025 20:04:19.903854 (+3377828us) ts_tablet_manager.cc:1151] Deleting tablet da8a0eaa68044835a4cf94340ca25ac5
1025 20:04:19.914508 (+ 10654us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 20:04:21.355575 (+1441067us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b54001c2ad4. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b41ef2 00007fd825524afc 00007fd826b33daa 00007fd826b64a10 00007fd826b4afdd 00007fd826b4af89 00007fd826b4af19 00007fd826b4aee1 00007fd826b4a10d 0000000000379254
1025 20:04:22.837841 (+1482266us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:23.231505 (+393664us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:23.248358 (+ 16853us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":885976,"spinlock_wait_cycles":85485440,"tablet-delete.queue_time_us":3370561}
I20251025 20:04:23.374064 32183 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:23.497239 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:23.509332 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:23.486338 32230 raft_consensus.cc:2749] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:23.372200 32131 consensus_queue.cc:237] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.493147 32082 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:23.501672 32093 raft_consensus.cc:3060] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:23.514293 30595 raft_consensus.cc:2468] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 5.
I20251025 20:04:23.515544 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06d7fa7a698c4890af235aed97fd6fb5" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:23.516402 30645 leader_election.cc:336] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.608s (SENT)
W20251025 20:04:23.523319 32238 log.cc:927] Time spent T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
W20251025 20:04:23.523711 32235 log.cc:927] Time spent T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
W20251025 20:04:23.524089 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:23.466918 32216 raft_consensus.cc:515] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.536736 30645 leader_election.cc:304] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:23.539865 30645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:23.540364 30645 leader_election.cc:336] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.642s (SENT)
I20251025 20:04:23.540306 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.541028 30645 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.459802 32189 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:23.546823 30645 leader_election.cc:336] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.659s (SENT)
I20251025 20:04:23.549590 30645 leader_election.cc:304] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.557471 32213 raft_consensus.cc:2749] T 0d23f4ba1a4b411baf57f59f30489265 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:23.558615 32023 tablet_replica.cc:333] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: stopping tablet replica
I20251025 20:04:23.559624 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f1269e0f1364a69b9a36ce877b0da5e" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.549314 32115 raft_consensus.cc:2749] T db5a3b726f79432baf8b25596709ca04 P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:23.534614 31993 consensus_queue.cc:237] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.465132 32100 rpcz_store.cc:276] 1025 20:04:18.909779 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:18.953105 (+ 43326us) service_pool.cc:225] Handling call
1025 20:04:22.043503 (+3090398us) ts_tablet_manager.cc:1151] Deleting tablet fea0c74c9c0c4a69afe072c45c4cbe99
1025 20:04:22.043566 (+ 63us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 20:04:22.887265 (+843699us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:23.064063 (+176798us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:23.362502 (+298439us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24107,"tablet-delete.queue_time_us":3064693}
I20251025 20:04:23.586292 32100 tablet_replica.cc:333] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: stopping tablet replica
I20251025 20:04:23.506325 32051 raft_consensus.cc:359] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:23.562904 32236 log.cc:927] Time spent T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20251025 20:04:23.545878 32072 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:23.601342 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 154659us
W20251025 20:04:23.564185 32240 log.cc:927] Time spent T ec9d5d5c538f4253bcec6b8b23843ed6 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20251025 20:04:23.598306 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.599752 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.600296 32207 raft_consensus.cc:2749] T 7904fa0f61dd4c9fadc39faddc8f908c P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251025 20:04:23.600854 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 145726us
I20251025 20:04:23.603669 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.595748 32051 raft_consensus.cc:385] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 20:04:23.603047 32237 log.cc:927] Time spent T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20251025 20:04:23.605906 30596 raft_consensus.cc:2393] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 4: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:23.606180 32142 consensus_queue.cc:1048] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20251025 20:04:23.606277 30486 catalog_manager.cc:4985] TS 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): tablet fea0c74c9c0c4a69afe072c45c4cbe99 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) successfully deleted
W20251025 20:04:23.632397 32244 log.cc:927] Time spent T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20251025 20:04:23.632678 30719 leader_election.cc:304] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:23.632561 31965 raft_consensus.cc:515] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.633878 30601 raft_consensus.cc:1240] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 5. Current term is 6. Ops: []
I20251025 20:04:23.634238 32144 raft_consensus.cc:2804] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:04:23.634831 32144 raft_consensus.cc:697] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [term 1 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:23.636564 30716 leader_election.cc:343] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:04:23.622854 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:23.688864 30745 raft_consensus.cc:2468] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 3.
W20251025 20:04:23.664300 30645 leader_election.cc:336] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.689s (SENT)
I20251025 20:04:23.693095 30645 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:23.685736 30644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:23.694504 32062 raft_consensus.cc:2749] T 54d399a886ee4166a2ea1b4410f9425a P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251025 20:04:23.694850 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 57330.5us
I20251025 20:04:23.695327 30489 catalog_manager.cc:4985] TS 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): tablet da8a0eaa68044835a4cf94340ca25ac5 (table test-workload [id=0622ca5c803d479d9858172423e767c7]) successfully deleted
I20251025 20:04:23.616756 32210 consensus_queue.cc:260] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.699059 32210 raft_consensus.cc:3060] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:23.699621 32142 consensus_queue.cc:1048] T 542d67f7d7544649be29769edeff1d0b P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:04:23.615932 32215 raft_consensus.cc:3060] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:23.607852 32189 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:23.692710 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "393c7e0e9757478198789a4dd6c9a8a8" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.611145 32216 leader_election.cc:290] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:23.622124 32220 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:23.706651 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:23.706459 32144 consensus_queue.cc:237] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.612262 32025 ts_tablet_manager.cc:1916] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:04:23.707770 30645 leader_election.cc:304] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:23.632674 30749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53762 (request call id 612) took 1537 ms. Trace:
I20251025 20:04:23.706576 32220 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.711483 32220 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:23.712170 30749 rpcz_store.cc:276] 1025 20:04:22.095116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:23.132774 (+1037658us) service_pool.cc:225] Handling call
1025 20:04:23.142410 (+ 9636us) raft_consensus.cc:1471] Updating replica for 2 ops
1025 20:04:23.166500 (+ 24090us) raft_consensus.cc:1534] Early marking committed up to index 0
1025 20:04:23.166541 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1025 20:04:23.179915 (+ 13374us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b2c000bb0d8. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8204399b1 00007fd820448fe2 00007fd821b75701 00007fd821b39fda 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22
1025 20:04:23.180500 (+ 585us) log.cc:844] Serialized 52 byte log entry
1025 20:04:23.262156 (+ 81656us) raft_consensus.cc:1645] Marking committed up to 4
1025 20:04:23.426696 (+164540us) log.cc:844] Serialized 14 byte log entry
1025 20:04:23.599038 (+172342us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b18002633e0. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd8278d09a1 00007fd8278dd371 00007fd81ea02caa 00007fd828074a29 00007fd81e8c2c30 00007fd81ea8aa94 00007fd81ea8aa05 00007fd821b31c00 00007fd821b442a6 00007fd821b54e17 00007fd821b54d98
1025 20:04:23.599589 (+ 551us) log.cc:844] Serialized 14 byte log entry
1025 20:04:23.599850 (+ 261us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:23.599969 (+ 119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 20:04:23.632451 (+ 32482us) raft_consensus.cc:1685] finished
1025 20:04:23.632474 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 20:04:23.632611 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31135,"prepare.queue_time_us":102757,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":101,"raft.queue_time_us":225171,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":245,"spinlock_wait_cycles":70894720,"thread_start_us":209922,"threads_started":2,"wal-append.queue_time_us":148541}
W20251025 20:04:23.715935 32245 log.cc:927] Time spent T 54d399a886ee4166a2ea1b4410f9425a P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.193s user 0.002s sys 0.000s
I20251025 20:04:23.717504 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.717545 32216 raft_consensus.cc:2804] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251025 20:04:23.718322 30604 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:23.720400 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.721107 30602 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 6.
I20251025 20:04:23.618290 30595 raft_consensus.cc:2376] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 583af532dff04e8ebba45013323d516d in term 5. Re-sending same reply.
I20251025 20:04:23.596354 32225 raft_consensus.cc:2749] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251025 20:04:23.729096 30645 leader_election.cc:336] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.693s (SENT)
I20251025 20:04:23.734742 30645 leader_election.cc:304] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.729023 30602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.731230 30596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:23.736096 30596 raft_consensus.cc:2393] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 6: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:23.733613 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.622202 32051 raft_consensus.cc:740] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Initialized, Role: FOLLOWER
I20251025 20:04:23.730437 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:23.739163 30676 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 6.
W20251025 20:04:23.623955 32206 consensus_meta.cc:287] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.518s user 0.009s sys 0.001s
I20251025 20:04:23.618060 32183 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:23.718281 32216 raft_consensus.cc:493] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:23.618129 32099 raft_consensus.cc:697] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:23.751217 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f2c3828d3614c1b97d3daf7d33c6537" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:23.751986 30746 raft_consensus.cc:1240] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20251025 20:04:23.753196 30645 raft_consensus.cc:2686] T fea0c74c9c0c4a69afe072c45c4cbe99 P 583af532dff04e8ebba45013323d516d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251025 20:04:23.753312 30748 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 8.
I20251025 20:04:23.728061 32072 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:23.761628 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:23.760397 30675 raft_consensus.cc:1240] T 580e75e8dcd949e39aff92ada50b40be P 583af532dff04e8ebba45013323d516d [term 5 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:23.764959 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cbb781b2e4441b4be9b53f7c0947525" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
W20251025 20:04:23.769122 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:23.616086 32124 raft_consensus.cc:359] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:23.764876 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 56680.9us
I20251025 20:04:23.616786 32095 raft_consensus.cc:359] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.771533 30716 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:23.783452 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.797034 30598 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:23.773222 30719 leader_election.cc:304] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:23.748924 32100 raft_consensus.cc:2243] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Raft consensus shutting down.
I20251025 20:04:23.752058 32216 raft_consensus.cc:3060] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:23.777359 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "531b2bbad426413898122c589cfa9780" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:23.771240 32184 raft_consensus.cc:493] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:23.814917 32098 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251025 20:04:23.770959 32177 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:23.799777 32207 consensus_queue.cc:1059] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20251025 20:04:23.834719 32023 raft_consensus.cc:2243] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Raft consensus shutting down.
W20251025 20:04:23.838366 32096 consensus_meta.cc:287] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e: Time spent flushing consensus metadata: real 0.530s user 0.011s sys 0.002s
I20251025 20:04:23.793479 31965 leader_election.cc:290] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
W20251025 20:04:23.867388 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 94256.3us
I20251025 20:04:23.879074 30676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:23.879392 30598 raft_consensus.cc:3055] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Stepping down as leader of term 2
I20251025 20:04:23.879861 30676 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:23.879925 30598 raft_consensus.cc:740] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:23.874226 30600 raft_consensus.cc:2393] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 6: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:23.883574 30749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:23.884435 30749 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:23.800631 32124 raft_consensus.cc:385] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:23.898461 32124 raft_consensus.cc:740] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583af532dff04e8ebba45013323d516d, State: Initialized, Role: FOLLOWER
I20251025 20:04:23.880999 30598 consensus_queue.cc:260] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.899324 32124 consensus_queue.cc:260] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.899874 30598 raft_consensus.cc:3060] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:23.901438 32124 ts_tablet_manager.cc:1434] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d: Time spent starting tablet: real 0.565s user 0.000s sys 0.015s
W20251025 20:04:23.902597 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 173531us
W20251025 20:04:23.904361 30645 leader_election.cc:336] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.694s (SENT)
I20251025 20:04:23.904763 30645 leader_election.cc:304] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.905505 32131 raft_consensus.cc:2749] T ec9d5d5c538f4253bcec6b8b23843ed6 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:23.803966 32095 raft_consensus.cc:385] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:04:23.906057 32095 raft_consensus.cc:740] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Initialized, Role: FOLLOWER
I20251025 20:04:23.906867 32095 consensus_queue.cc:260] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.909595 32095 ts_tablet_manager.cc:1434] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e: Time spent starting tablet: real 0.790s user 0.014s sys 0.004s
I20251025 20:04:23.875546 32184 raft_consensus.cc:515] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:23.861085 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:23.875118 32177 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.773931 32051 consensus_queue.cc:260] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.895161 32099 consensus_queue.cc:237] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.896356 30716 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:23.862353 32098 raft_consensus.cc:493] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:23.866158 32100 raft_consensus.cc:2272] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:04:23.883165 32206 raft_consensus.cc:493] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:23.901748 32124 ts_tablet_manager.cc:1467] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d: Tablet startup took 567ms
W20251025 20:04:23.918254 30645 leader_election.cc:336] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.717s (SENT)
I20251025 20:04:23.918161 30749 raft_consensus.cc:2468] T 0d9831caf5d44fc080590d41bd68a083 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:23.930217 30598 raft_consensus.cc:1275] T 393c7e0e9757478198789a4dd6c9a8a8 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251025 20:04:23.933548 30676 raft_consensus.cc:1275] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20251025 20:04:23.937758 30645 leader_election.cc:336] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.722s (SENT)
I20251025 20:04:23.938081 30676 pending_rounds.cc:85] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 2
I20251025 20:04:23.938531 30676 pending_rounds.cc:107] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251025 20:04:23.938256 30645 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.938877 30676 raft_consensus.cc:2889] T 78afc6e697b1482391aa5245bee3d881 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:04:23.928545 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d9831caf5d44fc080590d41bd68a083" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
W20251025 20:04:23.940351 30645 leader_election.cc:336] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.741s (SENT)
I20251025 20:04:23.940918 30645 leader_election.cc:304] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.891821 32121 raft_consensus.cc:2804] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:23.944494 30746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4360132dff42a18667801272f14797" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:23.953934 30750 raft_consensus.cc:1275] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:23.955122 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:23.956224 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a03f8a32da8499dbc0e2a426d214001" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
W20251025 20:04:23.961109 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 64887.8us
I20251025 20:04:23.970274 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:23.971282 30598 raft_consensus.cc:1275] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:23.943400 30645 leader_election.cc:304] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:23.946249 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97f7928c55054a2f809df6e4ee99800a" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:23.975651 30752 raft_consensus.cc:2393] T 97f7928c55054a2f809df6e4ee99800a P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 in current term 3: Already voted for candidate 783f8fd6d80141b7912f66a954a6f92e in this term.
W20251025 20:04:23.917873 32095 ts_tablet_manager.cc:1467] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e: Tablet startup took 816ms
I20251025 20:04:23.771586 32072 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.929723 32098 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Advancing to term 7
I20251025 20:04:23.941666 32206 raft_consensus.cc:515] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:23.987443 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b406b18ee0364eb5b4e35826fe7290be" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:23.987972 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375e703ada5b4ee2a37ea5e2a9c75376" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
W20251025 20:04:23.983450 30645 leader_election.cc:336] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Timed out: RequestConsensusVote RPC to 127.28.192.193:33717 timed out after 1.845s (SENT)
I20251025 20:04:23.989184 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:23.990092 30645 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:23.979915 32234 raft_consensus.cc:2804] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251025 20:04:23.994005 32234 raft_consensus.cc:493] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:23.994158 30604 raft_consensus.cc:1275] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:23.994415 32234 raft_consensus.cc:3060] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Advancing to term 8
I20251025 20:04:23.997782 32051 ts_tablet_manager.cc:1434] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957: Time spent starting tablet: real 0.757s user 0.013s sys 0.013s
W20251025 20:04:23.998178 32051 ts_tablet_manager.cc:1467] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957: Tablet startup took 757ms
W20251025 20:04:23.998445 32051 ts_tablet_manager.cc:1469] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957: Trace:
1025 20:04:16.930530 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.933333 (+ 2803us) service_pool.cc:225] Handling call
1025 20:04:17.026563 (+ 93230us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:17.026672 (+ 109us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:17.677616 (+650944us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:17.746920 (+ 69304us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:17.747432 (+ 512us) inbound_call.cc:173] Queueing success response
1025 20:04:22.044689 (+4297257us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1025 20:04:23.240386 (+1195697us) ts_tablet_manager.cc:1435] Starting tablet replica
1025 20:04:23.241041 (+ 655us) tablet_replica.cc:261] Starting instrumentation
1025 20:04:23.241686 (+ 645us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":2426,"tablet-open.queue_time_us":4297337}
I20251025 20:04:24.000774 32051 tablet_bootstrap.cc:492] T 5ee1ba6e04b74e1faf5c9819144cd740 P 349d66b543bf47249ac78154b908b957: Bootstrap starting.
I20251025 20:04:23.951208 32121 raft_consensus.cc:493] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 583af532dff04e8ebba45013323d516d)
W20251025 20:04:24.002060 30567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 67254.7us
I20251025 20:04:24.002576 32121 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:23.985210 32173 raft_consensus.cc:2804] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 20:04:24.005784 32173 raft_consensus.cc:493] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:24.006196 32173 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Advancing to term 5
W20251025 20:04:24.006358 30570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:24.006812 30570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:24.007854 32072 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:24.008785 32184 leader_election.cc:290] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.011566 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:24.012391 32177 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.015483 30673 raft_consensus.cc:1240] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 5. Current term is 6. Ops: []
I20251025 20:04:24.016961 30673 raft_consensus.cc:1275] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251025 20:04:24.018349 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:24.018661 30671 raft_consensus.cc:1275] T 13be59a31cda407db6fa50a400f37e1f P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251025 20:04:23.965759 32100 ts_tablet_manager.cc:1916] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:04:23.950484 32131 raft_consensus.cc:2749] T bc001e0fba604d68bfa98fc25d70bbbb P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:23.950172 32215 raft_consensus.cc:2749] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:23.968819 32189 raft_consensus.cc:3055] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Stepping down as leader of term 5
W20251025 20:04:24.031184 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 47758.4us
I20251025 20:04:24.024564 30752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db5a3b726f79432baf8b25596709ca04" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:24.038524 30674 raft_consensus.cc:1240] T 4f2c3828d3614c1b97d3daf7d33c6537 P 583af532dff04e8ebba45013323d516d [term 7 LEADER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 6. Current term is 7. Ops: []
W20251025 20:04:24.021871 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:24.065930 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f69f57fe417f492d9b82ed651b9d70c1" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:24.030366 30671 raft_consensus.cc:1240] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:24.019915 30673 raft_consensus.cc:2468] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
W20251025 20:04:24.032701 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 45797.7us
I20251025 20:04:23.913887 32082 raft_consensus.cc:515] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.048769 30747 raft_consensus.cc:2468] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 8.
I20251025 20:04:24.081692 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:24.082685 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:24.082918 30716 leader_election.cc:304] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:24.027930 30746 raft_consensus.cc:1275] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Refusing update from remote peer 349d66b543bf47249ac78154b908b957: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251025 20:04:24.062949 30645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:24.071566 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53eca06f855d4e699f10ab84a2ba0981" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.081435 32179 raft_consensus.cc:493] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:23.937167 32124 ts_tablet_manager.cc:1469] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d: Trace:
1025 20:04:19.666716 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:19.668356 (+ 1640us) service_pool.cc:225] Handling call
1025 20:04:19.831007 (+162651us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:19.831109 (+ 102us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:20.175910 (+344801us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:20.257498 (+ 81588us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:20.282547 (+ 25049us) inbound_call.cc:173] Queueing success response
1025 20:04:21.436377 (+1153830us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1025 20:04:23.336461 (+1900084us) ts_tablet_manager.cc:1435] Starting tablet replica
1025 20:04:23.560984 (+224523us) tablet_replica.cc:261] Starting instrumentation
1025 20:04:23.614267 (+ 53283us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":118,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"tablet-open.queue_time_us":1143799}
I20251025 20:04:24.100009 30750 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc001e0fba604d68bfa98fc25d70bbbb" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:24.100133 32179 raft_consensus.cc:515] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.100674 30597 raft_consensus.cc:1275] T 813ab494bf654127a41723bb256c1ede P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:24.101145 32124 tablet_bootstrap.cc:492] T 01d56eb384124dfcb1bcd35d77e89f0e P 583af532dff04e8ebba45013323d516d: Bootstrap starting.
I20251025 20:04:24.101564 30674 raft_consensus.cc:1240] T 0d9831caf5d44fc080590d41bd68a083 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:24.102169 30747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df036864f934f2d81add520300e9ed6" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:24.094648 32108 consensus_queue.cc:1059] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20251025 20:04:24.103120 32152 consensus_queue.cc:1059] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:04:24.104285 32152 raft_consensus.cc:3055] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Stepping down as leader of term 4
I20251025 20:04:24.104658 32152 raft_consensus.cc:740] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:24.102527 32179 leader_election.cc:290] T 5ee1ba6e04b74e1faf5c9819144cd740 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:24.105638 32070 consensus_queue.cc:1059] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251025 20:04:24.019558 32194 consensus_queue.cc:1048] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20251025 20:04:24.106678 32070 raft_consensus.cc:3055] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:24.107029 32070 raft_consensus.cc:740] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:24.108310 32108 consensus_queue.cc:1243] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer 583af532dff04e8ebba45013323d516d log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251025 20:04:24.108857 32231 raft_consensus.cc:3055] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 6 LEADER]: Stepping down as leader of term 6
I20251025 20:04:24.107821 32070 consensus_queue.cc:260] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.109345 32231 raft_consensus.cc:740] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:24.105532 32152 consensus_queue.cc:260] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.110054 32096 raft_consensus.cc:2804] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:04:24.110545 32096 raft_consensus.cc:493] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:24.110986 32096 raft_consensus.cc:3060] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:24.110450 32070 raft_consensus.cc:3060] T 0d9831caf5d44fc080590d41bd68a083 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:24.115357 32152 raft_consensus.cc:3060] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
@ 0x7fd81ec72dcd testing::UnitTest::Run() at ??:0
I20251025 20:04:24.118642 32177 consensus_queue.cc:1048] T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251025 20:04:24.121816 30569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:24.099220 30752 raft_consensus.cc:2468] T db5a3b726f79432baf8b25596709ca04 P 783f8fd6d80141b7912f66a954a6f92e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 2.
I20251025 20:04:24.101732 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.129956 30595 raft_consensus.cc:2468] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 2.
I20251025 20:04:24.031080 32189 raft_consensus.cc:740] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
W20251025 20:04:24.135768 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 48690.8us
I20251025 20:04:24.140029 30745 raft_consensus.cc:1240] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
@ 0x378fbc RUN_ALL_TESTS() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:?
I20251025 20:04:24.022442 31965 consensus_queue.cc:1059] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.102s
I20251025 20:04:24.019850 32093 raft_consensus.cc:515] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.066800 32210 raft_consensus.cc:2749] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:24.151553 30719 leader_election.cc:304] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 783f8fd6d80141b7912f66a954a6f92e; no voters: 349d66b543bf47249ac78154b908b957, 583af532dff04e8ebba45013323d516d
I20251025 20:04:24.035579 32132 consensus_queue.cc:1048] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.100s
@ 0x377b8d main at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:?
I20251025 20:04:24.165977 30748 raft_consensus.cc:2468] T b406b18ee0364eb5b4e35826fe7290be P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 6.
W20251025 20:04:24.173870 30645 leader_election.cc:343] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717): Not found: Tablet not found: e8c971a6d4854180b4d243fa79fbbc11
I20251025 20:04:24.174386 30645 leader_election.cc:304] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:24.170066 30570 connection.cc:232] RPC call timeout handler was delayed by -1.0343757450002613s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
I20251025 20:04:24.162220 32065 consensus_queue.cc:1048] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251025 20:04:24.177371 30570 connection.cc:232] RPC call timeout handler was delayed by -1.0289757450002526s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15100000000000002s for a response
I20251025 20:04:24.180411 30603 raft_consensus.cc:2393] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 5: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:24.180240 30570 leader_election.cc:304] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:24.183533 30645 raft_consensus.cc:2686] T e8c971a6d4854180b4d243fa79fbbc11 P 583af532dff04e8ebba45013323d516d: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251025 20:04:24.202884 30719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 51484.5us
I20251025 20:04:24.171659 30744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4403a496a85d4a3ebab45b2cb1af580e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:24.234766 32082 leader_election.cc:290] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:24.238581 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 64751.7us
I20251025 20:04:24.240815 30645 leader_election.cc:304] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.242317 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:24.242470 32131 consensus_queue.cc:1048] T 424773f793a4443da2db316842a1f3b5 P 583af532dff04e8ebba45013323d516d [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251025 20:04:24.243043 30670 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:24.243825 32131 raft_consensus.cc:2749] T da4360132dff42a18667801272f14797 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251025 20:04:24.244976 32256 log.cc:927] Time spent T ec9d5d5c538f4253bcec6b8b23843ed6 P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251025 20:04:24.252660 30645 leader_election.cc:304] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.282859 30645 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.217757 31809 raft_consensus.cc:2749] T 53eca06f855d4e699f10ab84a2ba0981 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251025 20:04:24.323160 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:24.287664 30670 raft_consensus.cc:2468] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:24.346762 30716 leader_election.cc:304] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:24.284494 32132 raft_consensus.cc:2749] T 78afc6e697b1482391aa5245bee3d881 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:24.175750 32231 consensus_queue.cc:260] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.279749 32261 raft_consensus.cc:493] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:24.339082 32210 raft_consensus.cc:2749] T b0a9871641274e83b2300f3ad8095add P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:24.341455 32019 raft_consensus.cc:493] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:24.202818 32189 consensus_queue.cc:260] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.345036 32062 raft_consensus.cc:3060] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
W20251025 20:04:24.369045 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 86226.9us
I20251025 20:04:24.245987 32168 consensus_queue.cc:1048] T 813ab494bf654127a41723bb256c1ede P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.105s
W20251025 20:04:24.377609 30568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:24.510432 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 161775us
I20251025 20:04:24.587781 30645 leader_election.cc:304] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:24.372061 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 191824us
I20251025 20:04:24.441373 30601 raft_consensus.cc:1240] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20251025 20:04:24.630875 30719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:24.589493 30598 raft_consensus.cc:1240] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Rejecting Update request from peer 783f8fd6d80141b7912f66a954a6f92e for earlier term 5. Current term is 6. Ops: []
I20251025 20:04:24.504186 30751 raft_consensus.cc:2468] T 375e703ada5b4ee2a37ea5e2a9c75376 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349d66b543bf47249ac78154b908b957 in term 4.
I20251025 20:04:24.478440 31414 raft_consensus.cc:493] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:24.636279 31414 raft_consensus.cc:515] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.600425 30595 raft_consensus.cc:1275] T a7d77f4dd14642b7bdb7299e84673c03 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251025 20:04:24.438665 32255 log.cc:927] Time spent T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957: Append to log took a long time: real 0.296s user 0.001s sys 0.000s
W20251025 20:04:24.462216 32246 log.cc:927] Time spent T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
W20251025 20:04:24.433112 32257 log.cc:927] Time spent T 2eddbfc136b841098915d8ea2d82956a P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.210s user 0.001s sys 0.000s
I20251025 20:04:24.468351 32098 raft_consensus.cc:515] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:24.652943 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 65218us
I20251025 20:04:24.654332 30645 leader_election.cc:304] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:24.432389 32260 log.cc:927] Time spent T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20251025 20:04:24.661927 30671 raft_consensus.cc:1275] T 719d97d4d3534235ad57ba5d3d50de74 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251025 20:04:24.662765 30675 raft_consensus.cc:1275] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20251025 20:04:24.663133 30675 pending_rounds.cc:85] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d: Aborting all ops after (but not including) 2
I20251025 20:04:24.663334 30675 pending_rounds.cc:107] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251025 20:04:24.663560 30675 raft_consensus.cc:2889] T a7d77f4dd14642b7bdb7299e84673c03 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251025 20:04:24.665150 32051 tablet_bootstrap.cc:700] T 5ee1ba6e04b74e1faf5c9819144cd740 P 349d66b543bf47249ac78154b908b957: Time spent opening tablet: real 0.176s user 0.005s sys 0.000s
I20251025 20:04:24.656109 32131 raft_consensus.cc:2749] T d4fa51030cac4f9ba59bc010c9eb0034 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:24.634620 32194 consensus_queue.cc:1048] T 78afc6e697b1482391aa5245bee3d881 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20251025 20:04:24.666555 30678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:24.667201 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:24.393466 32019 raft_consensus.cc:515] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.667992 30670 raft_consensus.cc:2393] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e in current term 6: Already voted for candidate 583af532dff04e8ebba45013323d516d in this term.
I20251025 20:04:24.668352 32225 raft_consensus.cc:2749] T 8e7a1b97c4aa4812a26b178e32476392 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:24.669384 32019 leader_election.cc:290] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:24.670758 32152 consensus_meta.cc:287] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957: Time spent flushing consensus metadata: real 0.553s user 0.010s sys 0.004s
I20251025 20:04:24.645674 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d7ed8359ea1484ca7ff724d58fa7a87" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:24.646620 30570 leader_election.cc:304] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:24.460408 32231 raft_consensus.cc:3060] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Advancing to term 7
W20251025 20:04:24.501212 30642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:24.497182 32023 pending_rounds.cc:70] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Trying to abort 1 pending ops.
I20251025 20:04:24.535591 32124 tablet_bootstrap.cc:654] T 01d56eb384124dfcb1bcd35d77e89f0e P 583af532dff04e8ebba45013323d516d: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:24.648689 31414 leader_election.cc:290] T 76be6727aa3a42939ff2cd45105b91f3 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.363461 31809 raft_consensus.cc:2804] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:24.514447 32025 ts_tablet_manager.cc:1929] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251025 20:04:24.640071 30502 catalog_manager.cc:5649] T 6cbb781b2e4441b4be9b53f7c0947525 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:24.417374 32132 raft_consensus.cc:493] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:24.673797 32051 tablet_bootstrap.cc:654] T 5ee1ba6e04b74e1faf5c9819144cd740 P 349d66b543bf47249ac78154b908b957: Neither blocks nor log segments found. Creating new log.
I20251025 20:04:24.660634 30645 leader_election.cc:304] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.438663 32108 raft_consensus.cc:3055] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 5 LEADER]: Stepping down as leader of term 5
W20251025 20:04:23.981281 32095 ts_tablet_manager.cc:1469] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e: Trace:
1025 20:04:17.011559 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:17.013842 (+ 2283us) service_pool.cc:225] Handling call
1025 20:04:17.075194 (+ 61352us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 20:04:17.100356 (+ 25162us) ts_tablet_manager.cc:682] Creating new metadata...
1025 20:04:17.645135 (+544779us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:18.476485 (+831350us) tablet_replica.cc:206] Creating consensus instance
1025 20:04:18.489148 (+ 12663us) inbound_call.cc:173] Queueing success response
1025 20:04:21.690519 (+3201371us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1025 20:04:23.122651 (+1432132us) ts_tablet_manager.cc:1435] Starting tablet replica
1025 20:04:23.562423 (+439772us) tablet_replica.cc:261] Starting instrumentation
1025 20:04:23.614804 (+ 52381us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":21293,"raft.queue_time_us":2236,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"tablet-open.queue_time_us":3142761}
W20251025 20:04:24.518095 32216 consensus_meta.cc:287] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d: Time spent flushing consensus metadata: real 0.679s user 0.011s sys 0.003s
I20251025 20:04:24.637789 32065 raft_consensus.cc:493] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:24.682403 32142 raft_consensus.cc:493] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:24.679179 30570 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:24.405112 32206 leader_election.cc:290] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.637373 32183 raft_consensus.cc:493] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:24.716444 32183 raft_consensus.cc:515] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:24.718271 30569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:24.678594 30501 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39172 (request call id 91) took 8846 ms. Trace:
I20251025 20:04:24.683079 30753 raft_consensus.cc:2368] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 6. Current term is 7.
I20251025 20:04:24.499387 32121 raft_consensus.cc:515] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.725054 32098 leader_election.cc:290] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:24.416226 32253 log.cc:927] Time spent T 542d67f7d7544649be29769edeff1d0b P 583af532dff04e8ebba45013323d516d: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20251025 20:04:24.682403 32023 pending_rounds.cc:77] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Aborting op as it isn't in flight: id { term: 3 index: 2 } timestamp: 7214787219220836352 op_type: NO_OP noop_request { }
I20251025 20:04:24.726471 32023 raft_consensus.cc:2889] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 20:04:24.726743 32121 leader_election.cc:290] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.726933 32023 raft_consensus.cc:2272] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251025 20:04:24.696486 32131 raft_consensus.cc:2749] T 5e47b11c8d47461cac220a7cc3625f58 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:24.709355 31809 raft_consensus.cc:697] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [term 4 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:24.735020 31809 consensus_queue.cc:237] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.737686 32199 raft_consensus.cc:493] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:24.738145 32023 ts_tablet_manager.cc:1916] T fea0c74c9c0c4a69afe072c45c4cbe99 P 349d66b543bf47249ac78154b908b957: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:04:24.738349 32199 raft_consensus.cc:515] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.740543 32199 leader_election.cc:290] T e4bb9e40b009470eb783f9bf63e0c3ba P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.741036 32199 raft_consensus.cc:493] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 20:04:24.743530 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 64426us
I20251025 20:04:24.463681 32093 leader_election.cc:290] T 542d67f7d7544649be29769edeff1d0b P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Requested vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:24.746359 32093 raft_consensus.cc:2764] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251025 20:04:24.684144 31965 raft_consensus.cc:2764] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251025 20:04:24.685604 30671 raft_consensus.cc:1240] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:24.685801 32132 raft_consensus.cc:515] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.749601 32132 leader_election.cc:290] T 8e7a1b97c4aa4812a26b178e32476392 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:24.687306 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "542d67f7d7544649be29769edeff1d0b" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:24.749857 32144 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:24.688542 30502 catalog_manager.cc:5649] T 4403a496a85d4a3ebab45b2cb1af580e P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 4 to 5, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 5 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:24.750490 32144 raft_consensus.cc:515] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.517158 32096 raft_consensus.cc:515] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.752688 32144 leader_election.cc:290] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.490367 32173 raft_consensus.cc:515] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
W20251025 20:04:24.692052 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251025 20:04:24.758997 30550 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251025 20:04:24.692965 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d986337b8c2047b4a1931044eb9cc540" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.694932 30645 leader_election.cc:304] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.760587 30595 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:24.393846 32261 raft_consensus.cc:515] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.419540 32234 raft_consensus.cc:515] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.657567 32070 consensus_queue.cc:1048] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20251025 20:04:24.684202 32025 log.cc:1199] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: Deleting WAL directory at /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-2-root/wals/8e7a1b97c4aa4812a26b178e32476392
I20251025 20:04:24.722041 32065 raft_consensus.cc:515] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.764631 30671 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e7a1b97c4aa4812a26b178e32476392" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:24.723556 32095 tablet_bootstrap.cc:492] T 2d6dd59f914c41758a18803991215caf P 783f8fd6d80141b7912f66a954a6f92e: Bootstrap starting.
I20251025 20:04:24.723902 32183 leader_election.cc:290] T 775d2ddde3b44056818240b2a8402e9c P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.751783 30502 catalog_manager.cc:5649] T da4360132dff42a18667801272f14797 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 4 to 5, leader changed from 349d66b543bf47249ac78154b908b957 (127.28.192.193) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 5 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:24.772235 32261 leader_election.cc:290] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 583af532dff04e8ebba45013323d516d (127.28.192.194:32975), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:24.771646 30502 catalog_manager.cc:5649] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 4 to 5, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 5 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:24.756909 30570 leader_election.cc:304] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
W20251025 20:04:24.761232 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 66402.5us
I20251025 20:04:24.774457 30645 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.775207 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:24.719079 30501 rpcz_store.cc:276] 1025 20:04:15.832649 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:15.832877 (+ 228us) service_pool.cc:225] Handling call
1025 20:04:24.678510 (+8845633us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 20:04:22.559657 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 20:04:22.560334 (+ 677us) write_op.cc:432] Acquiring schema lock in shared mode
1025 20:04:22.560392 (+ 58us) write_op.cc:435] Acquired schema lock
1025 20:04:22.560411 (+ 19us) tablet.cc:660] Decoding operations
1025 20:04:22.562135 (+ 1724us) write_op.cc:620] Acquiring the partition lock for write op
1025 20:04:22.562280 (+ 145us) write_op.cc:641] Partition lock acquired for write op
1025 20:04:22.562327 (+ 47us) tablet.cc:683] Acquiring locks for 15 operations
1025 20:04:22.564054 (+ 1727us) tablet.cc:699] Row locks acquired
1025 20:04:22.564087 (+ 33us) write_op.cc:260] PREPARE: finished
1025 20:04:22.564333 (+ 246us) write_op.cc:270] Start()
1025 20:04:22.564566 (+ 233us) write_op.cc:276] Timestamp: P: 1761422662564244 usec, L: 0
1025 20:04:22.564596 (+ 30us) op_driver.cc:348] REPLICATION: starting
1025 20:04:22.565642 (+ 1046us) log.cc:844] Serialized 6018 byte log entry
1025 20:04:23.095770 (+530128us) op_driver.cc:464] REPLICATION: finished
1025 20:04:23.288086 (+192316us) write_op.cc:301] APPLY: starting
1025 20:04:23.288180 (+ 94us) tablet.cc:1365] starting BulkCheckPresence
1025 20:04:23.288497 (+ 317us) tablet.cc:1368] finished BulkCheckPresence
1025 20:04:23.288620 (+ 123us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 20:04:23.627781 (+339161us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b5c00163e8c. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd827ce24f1 00007fd827ce23e1 00007fd827cdd12c 00007fd82547f1db 00007fd82547eea3 00007fd82557d5b4 00007fd8255690dc 00007fd82556d4c1 00007fd82556d479 00007fd82556d409 00007fd82556d3d1
1025 20:04:23.628399 (+ 618us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 20:04:23.628669 (+ 270us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
1025 20:04:23.628717 (+ 48us) write_op.cc:312] APPLY: finished
1025 20:04:23.629097 (+ 380us) log.cc:844] Serialized 106 byte log entry
1025 20:04:23.691735 (+ 62638us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:04:23.715233 (+ 23498us) write_op.cc:454] Released schema lock
1025 20:04:23.715298 (+ 65us) write_op.cc:341] FINISH: Updating metrics
1025 20:04:23.715766 (+ 468us) write_op.cc:489] Releasing partition, row and schema locks
1025 20:04:23.715804 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":918277,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":192005,"apply.run_cpu_time_us":9410,"apply.run_wall_time_us":427638,"num_ops":15,"prepare.queue_time_us":281760,"prepare.run_cpu_time_us":26815,"prepare.run_wall_time_us":116487,"raft.queue_time_us":159995,"raft.run_cpu_time_us":26783,"raft.run_wall_time_us":157273,"replication_time_us":530979,"spinlock_wait_cycles":7377792,"thread_start_us":619872,"threads_started":4,"wal-append.queue_time_us":140203,"wal-append.run_cpu_time_us":26335,"wal-append.run_wall_time_us":1927722}]]}
I20251025 20:04:24.775619 30645 leader_election.cc:304] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.741472 32199 raft_consensus.cc:515] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.754006 32216 raft_consensus.cc:515] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.773435 30502 catalog_manager.cc:5649] T 43d8dfe478a4430da93c214b20963f9f P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 2 to 3, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 3 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:24.695219 32108 raft_consensus.cc:740] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:24.778537 31414 consensus_queue.cc:1048] T 719d97d4d3534235ad57ba5d3d50de74 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251025 20:04:24.779227 30672 raft_consensus.cc:2410] T d539e97d6aea48aeb11438d95241d1bd P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251025 20:04:24.778745 30502 catalog_manager.cc:5649] T 375e703ada5b4ee2a37ea5e2a9c75376 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 1 to 3, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:24.783524 30673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:24.781270 30675 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:24.783572 30502 catalog_manager.cc:5649] T 4f2c3828d3614c1b97d3daf7d33c6537 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:24.786278 31414 consensus_queue.cc:1048] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:04:24.787828 30645 leader_election.cc:304] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.788380 31414 raft_consensus.cc:493] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:24.758496 32142 raft_consensus.cc:515] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.763885 32131 raft_consensus.cc:2749] T 4614bdb7f750431fa840cabccf782b4b P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:24.786684 30502 catalog_manager.cc:5649] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:24.797143 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:24.798234 30570 leader_election.cc:304] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.800755 30677 raft_consensus.cc:1240] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Rejecting Update request from peer 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4. Ops: []
I20251025 20:04:24.801385 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:24.802388 30672 raft_consensus.cc:2468] T b29cc29499b448adb358d6e30d3be8aa P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:24.766978 32206 raft_consensus.cc:493] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)
I20251025 20:04:24.791896 31414 raft_consensus.cc:515] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.805035 30570 leader_election.cc:304] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.791661 30645 leader_election.cc:304] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
W20251025 20:04:24.808359 30716 consensus_peers.cc:597] T 7294f698d29e4c919745b460e4e25eeb P 783f8fd6d80141b7912f66a954a6f92e -> Peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Couldn't send request to peer 583af532dff04e8ebba45013323d516d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:04:24.810361 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:24.813721 30679 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Advancing to term 7
I20251025 20:04:24.813939 30645 leader_election.cc:304] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.780793 32072 raft_consensus.cc:2749] T 2eddbfc136b841098915d8ea2d82956a P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:24.814487 30595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76be6727aa3a42939ff2cd45105b91f3" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.789736 32025 ts_tablet_manager.cc:1950] T 8e7a1b97c4aa4812a26b178e32476392 P 783f8fd6d80141b7912f66a954a6f92e: Deleting consensus metadata
I20251025 20:04:24.816288 30645 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.804977 30502 catalog_manager.cc:5649] T 5e47b11c8d47461cac220a7cc3625f58 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 3 to 4, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:24.824234 31414 leader_election.cc:290] T d4fa51030cac4f9ba59bc010c9eb0034 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.793864 32225 raft_consensus.cc:2749] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251025 20:04:24.794072 32096 leader_election.cc:290] T 1f781ca57ae9473b858d6d4d6bceeb53 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 3 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.805203 32093 raft_consensus.cc:2764] T 7294f698d29e4c919745b460e4e25eeb P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251025 20:04:24.830623 32261 raft_consensus.cc:2749] T 2eddbfc136b841098915d8ea2d82956a P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:24.839737 32234 leader_election.cc:290] T d1f9f57422164e5182f82b532e1bef47 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 8 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:24.851765 32065 leader_election.cc:290] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.833698 32108 consensus_queue.cc:260] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.861181 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "775d2ddde3b44056818240b2a8402e9c" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.826547 30570 leader_election.cc:400] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4.
I20251025 20:04:24.815407 32232 raft_consensus.cc:2749] T 54d399a886ee4166a2ea1b4410f9425a P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:24.794281 32179 raft_consensus.cc:2749] T 5f27e13dcf894a1487ed057731df6174 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:24.822355 32215 raft_consensus.cc:2749] T 4403a496a85d4a3ebab45b2cb1af580e P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:24.825093 32111 raft_consensus.cc:2749] T d4af0795c8ff48d9b639058ea7111f12 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251025 20:04:24.773298 31965 consensus_queue.cc:1059] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20251025 20:04:24.829954 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.862154 30570 leader_election.cc:403] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251025 20:04:24.863894 30679 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 7.
I20251025 20:04:24.825748 30601 raft_consensus.cc:1275] T 719d97d4d3534235ad57ba5d3d50de74 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251025 20:04:24.832999 30645 leader_election.cc:304] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.864956 30570 leader_election.cc:304] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.835810 32062 raft_consensus.cc:2749] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:24.837090 32216 leader_election.cc:290] T 1d5b049633ac4f21b7a8e34d536fd164 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541)
I20251025 20:04:24.838358 32173 leader_election.cc:290] T 27396efdc31841b68b8b2ea7ca531514 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 election: Requested vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:24.865772 32261 raft_consensus.cc:2749] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251025 20:04:24.819828 30502 catalog_manager.cc:5649] T 380df7888a29498d99ec3fe64410a542 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 4 to 6, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 6 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:24.866828 30678 raft_consensus.cc:2468] T d986337b8c2047b4a1931044eb9cc540 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 4.
I20251025 20:04:24.866209 30570 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.871376 30645 leader_election.cc:304] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.872321 30502 catalog_manager.cc:5649] T b406b18ee0364eb5b4e35826fe7290be P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:24.876399 32189 raft_consensus.cc:3060] T 4403a496a85d4a3ebab45b2cb1af580e P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:24.879245 30502 catalog_manager.cc:5649] T d33d554021204714a7a71332612e2070 P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:24.882521 30601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "424773f793a4443da2db316842a1f3b5" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:24.883108 30601 raft_consensus.cc:3060] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:24.882360 30570 leader_election.cc:304] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 349d66b543bf47249ac78154b908b957; no voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.882829 30716 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:24.884011 30502 catalog_manager.cc:5649] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 3 to 5, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 5 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:24.886695 30645 leader_election.cc:304] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.888131 30645 leader_election.cc:304] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.856016 32108 raft_consensus.cc:3060] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:24.898461 31965 raft_consensus.cc:3055] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Stepping down as leader of term 3
I20251025 20:04:24.898943 31965 raft_consensus.cc:740] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:24.899844 31965 consensus_queue.cc:260] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.901178 31965 raft_consensus.cc:3060] T 2d7ed8359ea1484ca7ff724d58fa7a87 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:24.903028 30570 leader_election.cc:304] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:24.903940 32268 raft_consensus.cc:2749] T 3f1269e0f1364a69b9a36ce877b0da5e P 349d66b543bf47249ac78154b908b957 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:24.904839 32268 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:24.905172 32093 consensus_queue.cc:1048] T 7904fa0f61dd4c9fadc39faddc8f908c P 349d66b543bf47249ac78154b908b957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251025 20:04:24.905442 32268 raft_consensus.cc:697] T b29cc29499b448adb358d6e30d3be8aa P 349d66b543bf47249ac78154b908b957 [term 5 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:24.885470 30502 catalog_manager.cc:5649] T d539e97d6aea48aeb11438d95241d1bd P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: UNKNOWN } } }
I20251025 20:04:24.863567 32131 raft_consensus.cc:3060] T 9cb15f2debdf450ab73ed74f6633fea7 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:24.890805 30677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41100 (request call id 657) took 1043 ms. Trace:
I20251025 20:04:24.891950 30645 leader_election.cc:304] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.874307 32199 leader_election.cc:290] T 01d56eb384124dfcb1bcd35d77e89f0e P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
W20251025 20:04:24.911654 30567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 20:04:24.924995 30677 rpcz_store.cc:276] 1025 20:04:23.847351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:23.847526 (+ 175us) service_pool.cc:225] Handling call
1025 20:04:23.847886 (+ 360us) raft_consensus.cc:1471] Updating replica for 0 ops
1025 20:04:24.801330 (+953444us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 20:04:24.890534 (+ 89204us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b540007c954. stack: 00007fd81ea5ba23 00007fd81ea5b920 00007fd81e3ae574 00007fd828055b14 00007fd828055aa9 00007fd821aa34c1 00007fd821b3a9a1 00007fd821b393de 00007fd826ae19d9 00007fd82007bd34 00007fd82007bcc2 00007fd82007bbf1 00007fd82007bb6c 00007fd82007ae22 00007fd81fe79ecc 00007fd81fe79306
1025 20:04:24.890743 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2190805376}
I20251025 20:04:24.906566 30570 leader_election.cc:304] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:24.927865 30645 leader_election.cc:304] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:24.929841 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01d56eb384124dfcb1bcd35d77e89f0e" candidate_uuid: "349d66b543bf47249ac78154b908b957" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:24.931021 30716 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:24.934065 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb9e40b009470eb783f9bf63e0c3ba" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.873271 32062 raft_consensus.cc:2749] T 76be6727aa3a42939ff2cd45105b91f3 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251025 20:04:24.935633 30597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc26ecef9f79461a932461d9c32a094f" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:24.939460 30597 raft_consensus.cc:3060] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Advancing to term 7
I20251025 20:04:24.866691 32152 raft_consensus.cc:2749] T 13be59a31cda407db6fa50a400f37e1f P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195:46541) with higher term. Message: Invalid argument: T 13be59a31cda407db6fa50a400f37e1f P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 349d66b543bf47249ac78154b908b957 for earlier term 3. Current term is 4.
I20251025 20:04:24.933679 30672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:24.944442 30672 raft_consensus.cc:3060] T 1f781ca57ae9473b858d6d4d6bceeb53 P 583af532dff04e8ebba45013323d516d [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:24.930960 30502 catalog_manager.cc:5649] T 20fe84e124304a7a9d92910cfa619beb P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 4 to 5, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 5 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
I20251025 20:04:24.940351 30570 leader_election.cc:304] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:24.823529 32206 raft_consensus.cc:515] T d33d554021204714a7a71332612e2070 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:24.941613 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b29cc29499b448adb358d6e30d3be8aa" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.943800 30603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f781ca57ae9473b858d6d4d6bceeb53" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:24.874112 32100 ts_tablet_manager.cc:1929] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.2
I20251025 20:04:24.871749 32261 raft_consensus.cc:2749] T d986337b8c2047b4a1931044eb9cc540 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251025 20:04:24.912853 32111 raft_consensus.cc:3060] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:24.964690 30677 raft_consensus.cc:1275] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Refusing update from remote peer 783f8fd6d80141b7912f66a954a6f92e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:24.945741 30502 catalog_manager.cc:5649] T b0a9871641274e83b2300f3ad8095add P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 3 to 4, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 4 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
W20251025 20:04:24.985999 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 45740.4us
I20251025 20:04:24.992746 30570 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e; no voters: 583af532dff04e8ebba45013323d516d
I20251025 20:04:24.945947 30603 raft_consensus.cc:3060] T 1f781ca57ae9473b858d6d4d6bceeb53 P 349d66b543bf47249ac78154b908b957 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:04:24.990408 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.994129 30600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957"
I20251025 20:04:24.994661 30600 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:24.889321 32194 raft_consensus.cc:493] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:24.992058 30502 catalog_manager.cc:5649] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 1 to 5, leader changed from 583af532dff04e8ebba45013323d516d (127.28.192.194) to 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195). New cstate: current_term: 5 leader_uuid: "783f8fd6d80141b7912f66a954a6f92e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } } }
W20251025 20:04:24.999647 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 71848.6us
I20251025 20:04:25.002002 30645 leader_election.cc:304] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:25.005326 32062 raft_consensus.cc:2749] T 1df036864f934f2d81add520300e9ed6 P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:25.006524 32062 raft_consensus.cc:2749] T dc26ecef9f79461a932461d9c32a094f P 583af532dff04e8ebba45013323d516d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251025 20:04:24.912407 32072 raft_consensus.cc:2749] T 06d7fa7a698c4890af235aed97fd6fb5 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 20:04:25.028589 30598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.912635 32215 raft_consensus.cc:2749] T 3f1269e0f1364a69b9a36ce877b0da5e P 583af532dff04e8ebba45013323d516d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:04:25.028172 30502 catalog_manager.cc:5649] T e4bb9e40b009470eb783f9bf63e0c3ba P 349d66b543bf47249ac78154b908b957 reported cstate change: term changed from 2 to 4, leader changed from 783f8fd6d80141b7912f66a954a6f92e (127.28.192.195) to 349d66b543bf47249ac78154b908b957 (127.28.192.193). New cstate: current_term: 4 leader_uuid: "349d66b543bf47249ac78154b908b957" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } health_report { overall_health: HEALTHY } } }
I20251025 20:04:25.029915 30598 raft_consensus.cc:2410] T 580e75e8dcd949e39aff92ada50b40be P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 783f8fd6d80141b7912f66a954a6f92e for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20251025 20:04:25.026974 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4fa51030cac4f9ba59bc010c9eb0034" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:25.042526 30601 raft_consensus.cc:2468] T 424773f793a4443da2db316842a1f3b5 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:24.946617 32070 raft_consensus.cc:2804] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 5 FOLLOWER]: Leader election won for term 5
I20251025 20:04:24.942157 32225 raft_consensus.cc:2749] T 3a03f8a32da8499dbc0e2a426d214001 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 20:04:25.047531 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01d56eb384124dfcb1bcd35d77e89f0e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:24.998198 32270 raft_consensus.cc:2804] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 4 FOLLOWER]: Leader election won for term 4
I20251025 20:04:24.942421 32025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53746 (request call id 62) took 8073 ms. Trace:
I20251025 20:04:24.989742 32100 log.cc:1199] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Deleting WAL directory at /tmp/dist-test-taskDQXBgW/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761422556872874-29443-0/minicluster-data/ts-1-root/wals/da8a0eaa68044835a4cf94340ca25ac5
I20251025 20:04:24.970944 32121 raft_consensus.cc:2804] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 20:04:25.050773 30604 raft_consensus.cc:2468] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 0.
W20251025 20:04:25.055248 30645 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.193:33717, user_credentials={real_user=slave}} blocked reactor thread for 53374.2us
W20251025 20:04:25.057217 30716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.192.194:32975, user_credentials={real_user=slave}} blocked reactor thread for 126285us
I20251025 20:04:25.062325 30679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27396efdc31841b68b8b2ea7ca531514" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d"
I20251025 20:04:25.063055 30679 raft_consensus.cc:3060] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Advancing to term 5
I20251025 20:04:25.065986 32121 raft_consensus.cc:493] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:04:25.066473 32121 raft_consensus.cc:3060] T b29cc29499b448adb358d6e30d3be8aa P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:25.067595 32124 tablet_bootstrap.cc:492] T 01d56eb384124dfcb1bcd35d77e89f0e P 583af532dff04e8ebba45013323d516d: No bootstrap required, opened a new log
I20251025 20:04:25.068174 32124 ts_tablet_manager.cc:1403] T 01d56eb384124dfcb1bcd35d77e89f0e P 583af532dff04e8ebba45013323d516d: Time spent bootstrapping tablet: real 0.967s user 0.024s sys 0.020s
I20251025 20:04:25.069630 32111 raft_consensus.cc:2749] T 375e703ada5b4ee2a37ea5e2a9c75376 P 583af532dff04e8ebba45013323d516d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 20:04:25.070106 32183 raft_consensus.cc:2804] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 7 FOLLOWER]: Leader election won for term 7
I20251025 20:04:25.070782 32100 ts_tablet_manager.cc:1950] T da8a0eaa68044835a4cf94340ca25ac5 P 583af532dff04e8ebba45013323d516d: Deleting consensus metadata
I20251025 20:04:25.061105 30645 leader_election.cc:304] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:25.072948 30645 leader_election.cc:304] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583af532dff04e8ebba45013323d516d; no voters: 349d66b543bf47249ac78154b908b957, 783f8fd6d80141b7912f66a954a6f92e
I20251025 20:04:25.081431 30677 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01d56eb384124dfcb1bcd35d77e89f0e" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:25.086077 30679 raft_consensus.cc:2468] T 27396efdc31841b68b8b2ea7ca531514 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 5.
I20251025 20:04:25.086537 30597 raft_consensus.cc:2468] T dc26ecef9f79461a932461d9c32a094f P 349d66b543bf47249ac78154b908b957 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 7.
I20251025 20:04:25.048720 32152 raft_consensus.cc:2804] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader election won for term 6
W20251025 20:04:25.082496 30567 leader_election.cc:343] T 01d56eb384124dfcb1bcd35d77e89f0e P 349d66b543bf47249ac78154b908b957 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 583af532dff04e8ebba45013323d516d (127.28.192.194:32975): Illegal state: must be running to vote when last-logged opid is not known
I20251025 20:04:25.088944 30670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "580e75e8dcd949e39aff92ada50b40be" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "583af532dff04e8ebba45013323d516d" is_pre_election: true
I20251025 20:04:25.092557 30751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ee1ba6e04b74e1faf5c9819144cd740" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:25.009263 32231 consensus_queue.cc:1059] T d4fa51030cac4f9ba59bc010c9eb0034 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20251025 20:04:25.099645 30716 leader_election.cc:304] T d986337b8c2047b4a1931044eb9cc540 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583af532dff04e8ebba45013323d516d, 783f8fd6d80141b7912f66a954a6f92e; no voters:
I20251025 20:04:25.025197 32194 raft_consensus.cc:515] T 531b2bbad426413898122c589cfa9780 P 783f8fd6d80141b7912f66a954a6f92e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 } } peers { permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 } } peers { permanent_uuid: "783f8fd6d80141b7912f66a954a6f92e" member_type: VOTER last_known_addr { host: "127.28.192.195" port: 46541 } }
I20251025 20:04:25.080377 32270 raft_consensus.cc:697] T 6973d3401da04486853abdd647a5795a P 349d66b543bf47249ac78154b908b957 [term 4 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:25.108825 30753 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d539e97d6aea48aeb11438d95241d1bd" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e" is_pre_election: true
I20251025 20:04:25.115258 30753 raft_consensus.cc:2468] T d539e97d6aea48aeb11438d95241d1bd P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583af532dff04e8ebba45013323d516d in term 4.
I20251025 20:04:25.109576 30604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cb15f2debdf450ab73ed74f6633fea7" candidate_uuid: "783f8fd6d80141b7912f66a954a6f92e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "349d66b543bf47249ac78154b908b957" is_pre_election: true
I20251025 20:04:25.072010 32070 raft_consensus.cc:697] T 0d23f4ba1a4b411baf57f59f30489265 P 349d66b543bf47249ac78154b908b957 [term 5 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:25.087299 32179 raft_consensus.cc:2749] T 6cbb781b2e4441b4be9b53f7c0947525 P 583af532dff04e8ebba45013323d516d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251025 20:04:25.094866 32111 raft_consensus.cc:3060] T 531b2bbad426413898122c589cfa9780 P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Advancing to term 6
I20251025 20:04:25.121712 30748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f9f57422164e5182f82b532e1bef47" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:25.121217 30749 raft_consensus.cc:1240] T 424773f793a4443da2db316842a1f3b5 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Rejecting Update request from peer 583af532dff04e8ebba45013323d516d for earlier term 4. Current term is 5. Ops: []
I20251025 20:04:25.117071 30604 raft_consensus.cc:2468] T 9cb15f2debdf450ab73ed74f6633fea7 P 349d66b543bf47249ac78154b908b957 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 3.
W20251025 20:04:25.125798 30570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.192.195:46541, user_credentials={real_user=slave}} blocked reactor thread for 98474.8us
I20251025 20:04:25.126117 30748 raft_consensus.cc:2393] T d1f9f57422164e5182f82b532e1bef47 P 783f8fd6d80141b7912f66a954a6f92e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 583af532dff04e8ebba45013323d516d in current term 8: Already voted for candidate 349d66b543bf47249ac78154b908b957 in this term.
I20251025 20:04:25.103089 32275 raft_consensus.cc:493] T 5e47b11c8d47461cac220a7cc3625f58 P 783f8fd6d80141b7912f66a954a6f92e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:25.095435 32183 raft_consensus.cc:697] T dc26ecef9f79461a932461d9c32a094f P 783f8fd6d80141b7912f66a954a6f92e [term 7 LEADER]: Becoming Leader. State: Replica: 783f8fd6d80141b7912f66a954a6f92e, State: Running, Role: LEADER
I20251025 20:04:25.106222 32072 raft_consensus.cc:493] T 97f7928c55054a2f809df6e4ee99800a P 583af532dff04e8ebba45013323d516d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:25.098896 32152 raft_consensus.cc:697] T 531b2bbad426413898122c589cfa9780 P 349d66b543bf47249ac78154b908b957 [term 6 LEADER]: Becoming Leader. State: Replica: 349d66b543bf47249ac78154b908b957, State: Running, Role: LEADER
I20251025 20:04:25.100474 32225 raft_consensus.cc:493] T 813ab494bf654127a41723bb256c1ede P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:25.105738 32163 raft_consensus.cc:493] T 0d23f4ba1a4b411baf57f59f30489265 P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:25.105846 32166 raft_consensus.cc:493] T 4f2c3828d3614c1b97d3daf7d33c6537 P 783f8fd6d80141b7912f66a954a6f92e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:04:25.131413 30745 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d5b049633ac4f21b7a8e34d536fd164" candidate_uuid: "583af532dff04e8ebba45013323d516d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "783f8fd6d80141b7912f66a954a6f92e"
I20251025 20:04:25.135453 30745 raft_consensus.cc:3060] T 1d5b049633ac4f21b7a8e34d536fd164 P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Advancing to term 4
I20251025 20:04:25.089360 32065 consensus_queue.cc:1048] T a7d77f4dd14642b7bdb7299e84673c03 P 783f8fd6d80141b7912f66a954a6f92e [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d66b543bf47249ac78154b908b957" member_type: VOTER last_known_addr { host: "127.28.192.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20251025 20:04:25.112224 32142 leader_election.cc:290] T 9cb15f2debdf450ab73ed74f6633fea7 P 783f8fd6d80141b7912f66a954a6f92e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 349d66b543bf47249ac78154b908b957 (127.28.192.193:33717), 583af532dff04e8ebba45013323d516d (127.28.192.194:32975)
I20251025 20:04:25.110090 32215 raft_consensus.cc:493] T b406b18ee0364eb5b4e35826fe7290be P 583af532dff04e8ebba45013323d516d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 783f8fd6d80141b7912f66a954a6f92e)
I20251025 20:04:25.137120 30753 raft_consensus.cc:1275] T 580e75e8dcd949e39aff92ada50b40be P 783f8fd6d80141b7912f66a954a6f92e [term 5 FOLLOWER]: Refusing update from remote peer 583af532dff04e8ebba45013323d516d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251025 20:04:25.141645 30595 raft_consensus.cc:2468] T 76be6727aa3a42939ff2cd45105b91f3 P 349d66b543bf47249ac78154b908b957 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783f8fd6d80141b7912f66a954a6f92e in term 6.
I20251025 20:04:25.098652 32093 consensus_queue.cc:1059] T 1d5b049633ac4f21b7a8e34d536fd164 P 349d66b543bf47249ac78154b908b957 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "583af532dff04e8ebba45013323d516d" member_type: VOTER last_known_addr { host: "127.28.192.194" port: 32975 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
I20251025 20:04:25.056677 32025 rpcz_store.cc:276] 1025 20:04:16.862353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 20:04:16.872953 (+ 10600us) service_pool.cc:225] Handling call
1025 20:04:23.237526 (+6364573us) ts_tablet_manager.cc:1151] Deleting tablet 8e7a1b97c4aa4812a26b178e32476392
1025 20:04:23.237604 (+ 78us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 20:04:24.344080 (+1106476us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:24.514012 (+169932us) tablet_metadata.cc:641] Metadata flushed
1025 20:04:24.935272 (+421260us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":166421,"tablet-delete.queue_time_us":6363818}
@ 0x7fd81b323bf7 __libc_start_main at ??:0
@ 0x29847a _start at ??:?
I20251025 20:04:25.116057 32098 raft_consensus.cc:493] T 43d8dfe478a4430da93c214b20963f9f P 783f8fd6d80141b7912f66a954a6f92e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 349d66b543bf47249ac78154b908b957)